feat: add cmd to generate a JSON representation of schema #5919

sgulseth · 2024-03-06T13:57:53Z

Description

Adds a new command that can be executed within a sanity context to extract a JSON representation of the studios schema. This makes it easier for other tools(TM) to consume it without depending on browser environments etc. I tried to base most of the logic on sanity schema validate to keep consistency in the repo/package.

What to review

Correctness: Do we parse the schema correctly?
Naming things: It's hard! Can we name the command line argument or flags differently?
Note that we consider this feature beta, anything more than in the help text it should be indicated?

Testing

$ pnpm build
$ sanity schema extract # somewhere there's a sanity.config.ts

Notes for release

Beta: Extract sanity schema as JSON, to be consumed by other tools

vercel · 2024-03-06T13:58:00Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
performance-studio	✅ Ready (Inspect)	Visit Preview		Mar 12, 2024 10:44am
test-studio	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Mar 12, 2024 10:44am

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
studio-workshop	⬜️ Ignored (Inspect)	Visit Preview		Mar 12, 2024 10:44am

github-actions · 2024-03-06T14:01:36Z

No changes to documentation

github-actions · 2024-03-06T14:06:44Z

Component Testing Report Updated Mar 12, 2024 10:48 AM (UTC)

File	Status	Duration	Passed	Skipped
comments/CommentInput.spec.tsx	✅ Passed (Inspect)	32s	15	0
formBuilder/ArrayInput.spec.tsx	✅ Passed (Inspect)	6s	3	0
formBuilder/inputs/PortableText/Annotations.spec.tsx	✅ Passed (Inspect)	11s	3	0
formBuilder/inputs/PortableText/copyPaste/CopyPaste.spec.tsx	✅ Passed (Inspect)	12s	4	2
formBuilder/inputs/PortableText/Decorators.spec.tsx	✅ Passed (Inspect)	12s	6	0
formBuilder/inputs/PortableText/FocusTracking.spec.tsx	✅ Passed (Inspect)	31s	15	0
formBuilder/inputs/PortableText/Input.spec.tsx	✅ Passed (Inspect)	1m 0s	15	0
formBuilder/inputs/PortableText/ObjectBlock.spec.tsx	✅ Passed (Inspect)	1m 1s	18	0
formBuilder/inputs/PortableText/RangeDecoration.spec.tsx	✅ Passed (Inspect)	11s	6	0
formBuilder/inputs/PortableText/Styles.spec.tsx	✅ Passed (Inspect)	13s	6	0
formBuilder/inputs/PortableText/Toolbar.spec.tsx	✅ Passed (Inspect)	20s	9	0

socket-security · 2024-03-06T14:18:55Z

New dependencies detected. Learn more about Socket for GitHub ↗︎

Package	New capabilities	Transitives	Size	Publisher

View full report↗︎

rexxars

I'll leave the bulk of the review up to the tagged reviewers, but I have some thoughts/questions:

A bit reluctant to tie @sanity/schema to groq-js
- A schema extract command makes total sense to me, but the resulting format should preferably be generic enough that it can be used to generate all kinds of things; GraphQL types, TypeScript types, tailored API clients etc.
- Generally I would favor using a more established output type, say JSONSchema, as long as it is able to represent our schema completely.
- Perhaps we could introduce a format parameter/flag? We don't have to implement any other formats right now, but I think requiring the flag for now makes it more future-compatible if we want to default to (say) JSONSchema in the future.
Using the compiled schema (Schema.compile()/createSchema()) would be safer
- We shouldn't have to special-case image/file fields/types etc - and defining their properties are already done for you with this approach. It feels brittle/unnecessary to have to maintain two code paths if we add new fields.
- The generated type output may be missing types such as image and file asset documents, geopoints, slugs and other non-user-defined types
- Block types shouldn't have to be special-cased - they (should) just be an object type like any other. Given you can create named block types, this is important - as the name property may not be block
- Without having looked deeply at the code, I am curious if the current approach handles aliased types, or if that is left to whatever acts on these types to figure out

packages/@sanity/schema/package.json

packages/sanity/src/_internal/cli/actions/schema/extractAction.ts

packages/sanity/src/_internal/cli/commands/schema/extractSchemaCommand.ts

rexxars · 2024-03-07T16:11:10Z

packages/sanity/src/_internal/cli/commands/schema/extractSchemaCommand.ts

+Options
+  --workspace <name> The name of the workspace to generate a schema for
+  --path Optional path to specify destination of the schema file
+  --enforce-required-fields Makes the schema generated respect required fields in the schema. This can be used when generating types if you are only querying for published documents. Defaults to false.


This seems broader than just for querying - I would phrase this to be "treat fields marked as required as non-optional", as what it impacts is the resulting optional attributes

"querying for published documents" doesn't really encompass this - both drafts and published documents can be incomplete/not follow schema (documents that were published prior to a field being introduced, for instance)

This seems broader than just for querying - I would phrase this to be "treat fields marked as required as non-optional", as what it impacts is the resulting optional attributes

True, and agree on your phrasing 👍

This seems broader than just for querying - I would phrase this to be "treat fields marked as required as non-optional", as what it impacts is the resulting optional attributes
"querying for published documents" doesn't really encompass this - both drafts and published documents can be incomplete/not follow schema (documents that were published prior to a field being introduced, for instance).

Yeah, we do however consider the schema the source of truth so if anything diverges from your schema we don't give any guarantees with the types at least 🤔

packages/sanity/src/_internal/cli/commands/schema/extractSchemaCommand.ts

packages/sanity/src/_internal/cli/threads/extractSchema.ts

socket-security · 2024-03-08T05:53:52Z

👍 Dependency issues cleared. Learn more about Socket for GitHub ↗︎

This PR previously contained dependency changes with security issues that have been resolved, removed, or ignored.

View full report↗︎

sgulseth · 2024-03-08T08:19:32Z

A bit reluctant to tie @sanity/schema to groq-js.

Where would you prefer these types to live?

I don't see any reasons this couldn't be used with our graphql generator as well, would potentially need to extend ObjectAttribute to include more data, but that's doable.

We did talk about using jsonschema, but the ecosystem is a bit fragmented with different specification and support, so we opted for our own implementation here.

cc. @judofyr for additional feedback.

Perhaps we could introduce a format parameter/flag

We could. Would then this be a required flag to set? ie sanity schema extract --format=typenode Could it default to typenode at least while being beta? To simplify DX and feedback for now.

Using the compiled schema (Schema.compile()/createSchema()) would be safer

Ah, nice. I'll look into this!

Without having looked deeply at the code, I am curious if the current approach handles aliased types, or if that is left to whatever acts on these types to figure out

It does, they are referred to as InlineTypeNode since the value are inlined in Content Lake

bjoerge · 2024-03-08T10:50:35Z

A bit reluctant to tie @sanity/schema to groq-js.

Where would you prefer these types to live?

Any reason extractSchema couldn't live with the cli command for now?

packages/sanity/src/_internal/cli/threads/extractSchema.ts

sgulseth · 2024-03-08T15:15:36Z

@bjoerge then we either need to have the types in two places, guess that works with how ts infers types. Or groq-js would need to depend on the cli, which isn't ideal either

rexxars · 2024-03-08T18:23:19Z

Where would you prefer these types to live?

Not sure, honestly - probably fine for now.

We did talk about using jsonschema, but the ecosystem is a bit fragmented with different specification and support, so we opted for our own implementation here.

Would be interested in learning more about this, but I don't need a full write-up here - if there's any TLDR or links I can use to understand more that'd be great.

Perhaps we could introduce a format parameter/flag

We could. Would then this be a required flag to set? ie sanity schema extract --format=typenode Could it default to typenode at least while being beta? To simplify DX and feedback for now.

Ideally I would default to something that is a defined standard, such as JSONSchema, and make other formats a flag. I understand that it's an improved DX when we only have a single format, but it would also be a worse DX when we change the default (breaking change).

sgulseth · 2024-03-11T10:28:45Z

I have adressed the feedback and we are now processing the compiled types. OK to move the format discussion to slack?

rexxars

Overall this looks good to me, but would love if @bjoerge who knows the schema better could have a look.

One curiosity; seems to me like Portable Text block and span types do not have a _type property? Span should always be span (the type cannot be changed in any way), and the block type should either be block or the name given by the user.

bjoerge

Great work! Left a few comments for your consideration, only nits/nice-to-haves, so good to merge from my side!

packages/@sanity/schema/package.json

packages/@sanity/schema/src/sanity/extractSchema.ts

packages/@sanity/schema/test/extractSchema/extractSchema.test.ts

packages/@sanity/schema/test/extractSchema/fixtures/block.ts

* feat: add cmd to generate a JSON representation of schema * feat: refactor to use compiled schema * fix: append _type to objects * chore(schema): clean up and describe execution * chore(schema): lock groq-js to minor canary

sgulseth force-pushed the feat/schema-generation branch from 39b0521 to ef0c637 Compare March 6, 2024 14:17

vercel bot deployed to Preview – performance-studio March 6, 2024 14:20 View deployment

vercel bot deployed to Preview – test-studio March 6, 2024 14:21 View deployment

sgulseth marked this pull request as ready for review March 6, 2024 14:31

sgulseth requested a review from a team as a code owner March 6, 2024 14:31

sgulseth requested review from binoy14, bjoerge and ricokahler and removed request for a team and binoy14 March 6, 2024 14:31

sgulseth force-pushed the feat/schema-generation branch from ef0c637 to 494c46a Compare March 6, 2024 18:35

vercel bot deployed to Preview – performance-studio March 6, 2024 18:39 View deployment

vercel bot deployed to Preview – test-studio March 6, 2024 18:39 View deployment

rexxars reviewed Mar 7, 2024

View reviewed changes

sgulseth force-pushed the feat/schema-generation branch from 494c46a to 0d38a52 Compare March 8, 2024 08:26

vercel bot deployed to Preview – performance-studio March 8, 2024 08:30 View deployment

vercel bot deployed to Preview – test-studio March 8, 2024 08:30 View deployment

bjoerge reviewed Mar 8, 2024

View reviewed changes

packages/sanity/src/_internal/cli/threads/extractSchema.ts Outdated Show resolved Hide resolved

sgulseth force-pushed the feat/schema-generation branch from 0fcd3cf to b2f30bb Compare March 11, 2024 09:11

vercel bot deployed to Preview – performance-studio March 11, 2024 09:16 View deployment

vercel bot deployed to Preview – test-studio March 11, 2024 09:17 View deployment

sgulseth force-pushed the feat/schema-generation branch from b2f30bb to 86f94d4 Compare March 11, 2024 09:24

vercel bot deployed to Preview – performance-studio March 11, 2024 09:25 View deployment

vercel bot deployed to Preview – test-studio March 11, 2024 09:26 View deployment

sgulseth force-pushed the feat/schema-generation branch from 86f94d4 to 4afe4ab Compare March 11, 2024 10:16

vercel bot deployed to Preview – performance-studio March 11, 2024 10:20 View deployment

vercel bot deployed to Preview – test-studio March 11, 2024 10:21 View deployment

sgulseth requested review from rexxars and bjoerge March 11, 2024 10:27

rexxars approved these changes Mar 11, 2024

View reviewed changes

vercel bot deployed to Preview – performance-studio March 12, 2024 08:27 View deployment

vercel bot deployed to Preview – test-studio March 12, 2024 08:28 View deployment

bjoerge approved these changes Mar 12, 2024

View reviewed changes

vercel bot deployed to Preview – performance-studio March 12, 2024 10:20 View deployment

vercel bot deployed to Preview – test-studio March 12, 2024 10:21 View deployment

vercel bot deployed to Preview – performance-studio March 12, 2024 10:28 View deployment

vercel bot deployed to Preview – test-studio March 12, 2024 10:28 View deployment

sgulseth added 5 commits March 12, 2024 11:39

feat: add cmd to generate a JSON representation of schema

c8f7e32

feat: refactor to use compiled schema

b94b2b4

fix: append _type to objects

7d2d714

chore(schema): clean up and describe execution

4f13425

chore(schema): lock groq-js to minor canary

b9dd36d

sgulseth force-pushed the feat/schema-generation branch from 75d0de2 to b9dd36d Compare March 12, 2024 10:39

vercel bot deployed to Preview – performance-studio March 12, 2024 10:43 View deployment

vercel bot deployed to Preview – test-studio March 12, 2024 10:44 View deployment

sgulseth added this pull request to the merge queue Mar 12, 2024

Merged via the queue into next with commit c1e4f2a Mar 12, 2024
37 checks passed

sgulseth deleted the feat/schema-generation branch March 12, 2024 10:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add cmd to generate a JSON representation of schema #5919

feat: add cmd to generate a JSON representation of schema #5919

sgulseth commented Mar 6, 2024

vercel bot commented Mar 6, 2024 •

edited

github-actions bot commented Mar 6, 2024

github-actions bot commented Mar 6, 2024 •

edited

socket-security bot commented Mar 6, 2024 •

edited

rexxars left a comment

rexxars Mar 7, 2024

sgulseth Mar 8, 2024

socket-security bot commented Mar 8, 2024 •

edited

sgulseth commented Mar 8, 2024 •

edited by rexxars

bjoerge commented Mar 8, 2024

sgulseth commented Mar 8, 2024 •

edited

rexxars commented Mar 8, 2024

sgulseth commented Mar 11, 2024

rexxars left a comment

bjoerge left a comment

feat: add cmd to generate a JSON representation of schema #5919

feat: add cmd to generate a JSON representation of schema #5919

Conversation

sgulseth commented Mar 6, 2024

Description

What to review

Testing

Notes for release

vercel bot commented Mar 6, 2024 • edited

github-actions bot commented Mar 6, 2024

github-actions bot commented Mar 6, 2024 • edited

socket-security bot commented Mar 6, 2024 • edited

rexxars left a comment

Choose a reason for hiding this comment

rexxars Mar 7, 2024

Choose a reason for hiding this comment

sgulseth Mar 8, 2024

Choose a reason for hiding this comment

socket-security bot commented Mar 8, 2024 • edited

sgulseth commented Mar 8, 2024 • edited by rexxars

bjoerge commented Mar 8, 2024

sgulseth commented Mar 8, 2024 • edited

rexxars commented Mar 8, 2024

sgulseth commented Mar 11, 2024

rexxars left a comment

Choose a reason for hiding this comment

bjoerge left a comment

Choose a reason for hiding this comment

vercel bot commented Mar 6, 2024 •

edited

github-actions bot commented Mar 6, 2024 •

edited

socket-security bot commented Mar 6, 2024 •

edited

socket-security bot commented Mar 8, 2024 •

edited

sgulseth commented Mar 8, 2024 •

edited by rexxars

sgulseth commented Mar 8, 2024 •

edited