[RFC] Indexer API

[shilman]

Status: accepted
Tracked in: #23457

Summary

This proposes a new API for indexing stories and documents in Storybook. This makes it possible to support alternative syntaxes for stories and documents. For example, if somebody from the community wanted to support story authoring as Vue SFCs or docs in Markdown or Markdoc.

Problem Statement

Component Story Format (CSF) is a portable standard for capturing UI component examples. It is based on ECMAScript Modules (ESM), where the default export contains metadata about the component being documented, and each named export captures a key state of that component.

CSF has been incredibly successful in Storybook and has even been adopted by other tools in the frontend ecosystem. However, it has some key criticisms as well:

1. Static. Because ESM named exports are statically defined, it is not easy to dynamically generate stories in CSF. This make it hard to do combinatorial testing on component props, or automatically generate stories from static fixture data (both of which were fairly easy to do with the legacy storiesOf API that CSF replaced).
2. Fixed syntax. Renderers like Vue and Svelte have their own syntaxes for defining and using components, and some users would prefer to write their stories using these alternative syntaxes. Similarly, Storybook supports MDX for long-form documentation, but in the future might want to support pure Markdown or Markdoc instead.

We want an API that makes it possible to solve these problems in a flexible way.

Non-goals

1. Not a storiesOf replacement. We are not trying to solve storiesOf backwards compatibility. We are betting on static analysis and the solution described here is a “load-time” solution. We may provide a more dynamic solution in the future for dynamic “run-time” generating variants within a story and handling them sensibly across our tools.
2. Not an addon. We are not building addons for fixture-based stories, combinatorial stories, or alternative story syntaxes. We may do so in the future, but we want to make it possible for the community to experiment with options.
3. No loader/plugin tools. The solution described here assumes that the user’s story files are importable as ESM, which may require a “load-time” transformation such as a Vite plugin or Webpack loader. This proposals does not try to solve that problem, though our documentation will include examples for how to do this.

Implementation

To solve these problems, we propose a public indexer API for Storybook. The API would make it possible to extend Storybook in a variety of ways.

Existing architecture

Before introducing the API, let’s review Storybook’s architecture:

Config. Users define the files to index in .storybook/main.js. (The field is called stories, but should be renamed to index since it can contain docs as well, and potentially other types of entries in the future):

export default {
  stories: ['../src/**/*.mdx', '../src/**/*.stories.js'],
};

Index. The index is a global listing of all the entries in your storybook. It’s an in-memory data structure that gets updated dynamically as the user updates the file system, and is also served at the URL endpoint /index.json in v7 storybooks. It also defines what appears in Storybook’s sidebar.

Each file results in one or more entries in the index and in the sidebar. For example, consider this button example and its corresponding index.json entry and sidebar visualization.

// Button.stories.js
import { Button } from './Button';

export default {
  title: 'Example/Button',
  component: Button
};
export const Primary = { args: { primary: true } };
export const Secondary = { };

// index.json
{
  "example-button--primary": {
    "id": "example-button--primary",
    "title": "Example/Button",
    "name": "Primary",
    "importPath": "./src/stories/Button.stories.ts",
    "tags": ["story"],
    "type": "story"
  },
  "example-button--secondary": {
    "id": "example-button--secondary",
    "title": "Example/Button",
    "name": "Secondary",
    "importPath": "./src/stories/Button.stories.ts",
    "tags": ["story"],
    "type": "story"
  }
}

Implementation

The indexer is a function that generates zero or more index entries from an input file.

// type ExportName/Path/ComponentTitle/StoryName/StoryId/Tag = string;

export interface IndexerOptions {
  makeTitle: (userTitle?: string) => string;
}

/**
 * An indexer describes which filenames it handles, and how to index each individual file - turning it into an entry in the index.
 */
export type Indexer = {
  /**
   * A regular expression that should match all files to be handled by this indexer
   */
  test: RegExp;
  /**
   * Indexes a file containing stories or docs.
   * @param fileName The name of the file to index.
   * @param options {@link IndexerOptions} for indexing the file.
   * @returns A promise that resolves to an array of {@link IndexInput} objects.
   */
  createIndex: (fileName: string, options: IndexerOptions) => Promise<IndexInput[]>;
};

/**
 * The base input for indexing a story or docs entry.
 */
export type IndexInput = {
  type: 'story';
  /** The file to import from e.g. the story file. */
  importPath: Path;
  /** The name of the export to import. */
  exportName: ExportName;
  /** The name of the entry, auto-generated from {@link exportName} if unspecified. */
  name?: StoryName;
  /** The location in the sidebar, auto-generated from {@link importPath} if unspecified. */
  title?: ComponentTitle;
  /**
   * The custom id optionally set at `meta.id` if it needs to differ from the id generated via {@link title}.
   * If unspecified, the meta id will be auto-generated from {@link title}.
   * If specified, the meta in the CSF file _must_ have a matching id set at `meta.id`, to be correctly matched.
   */
  metaId?: MetaId;
  /** Tags for filtering entries in Storybook and its tools. */
  tags?: Tag[];
  /**
   * The id of the entry, auto-generated from {@link title}/{@link metaId} and {@link exportName} if unspecified.
   * If specified, the story in the CSF file _must_ have a matching id set at `parameters.__id`, to be correctly matched.
   * Only use this if you need to override the auto-generated id.
   */
  __id?: StoryId;
};

Note that this API only covers how to add story/docs entries to the index. It does not specify how to load those files. For that, your addon or preset would need to provide some other kind of configuration, such as a webpack/vite plugin, to load the importPath using whatever builder the user has configured in their project. For example, if you are creating an addon for writing stories as Vue SFCs, the indexer will tell Storybook the list of stories available in each Vue SFC story file, and the addon will provide a vite and webpack plugin (possibly using unplugin for convenience) to transform those SFCs into CSF at load time. We will elaborate on this more as needed.

Users can register the indexers in their .storybook/main.js config file and addons can register indexers using the presets API (which is the same as main.js).

interface StorybookConfig {
  indexers: Indexer[] | async (existing: Indexer[], options: Options) => Promise<Indexer[]>
}

Examples

Fixture data

Consider the following addon that generates stories for components based on JSON fixture data. It looks for jsonstories.js files in the project and then for each such file, looks for .json files in the same directory, each of which is added as a story.

import fs from 'fs/promises';
import path from 'path';

const JsonIndexer: Indexer = {
  test: /jsonstories.[tj]sx?$/,
  createIndex: async (fileName) => {
    // JSON files in the current directory
    const jsonFiles = (await fs.readdir(path.dirname(fileName))
      .filter(f => f.endsWith('.json'))

    // Sidebar entries for each file
    return jsonFiles
      .map(jsonFile => ({
        type: 'story',
        importPath: fileName,
        exportName: f.split('.')[0]
      });
  }
}

export default { indexers: [JsonIndexer] };

Combinatorial testing

Consider the following addon that supports combinatorial testing via an extra parameter combos in the default export which specifies the arg values to permute. It generates extra named exports of the form Combo0 ... ComboN for each combination of args.

import fs from 'fs/promises';
import path from 'path';

const CombosIndexer: Indexer = {
  test: /.combos.[tj]sx?$/,
  createIndex: async (fileName) => {
    const code = await fs.readFile(fileName);
    const combos = parseCombos(code);

    // Sidebar entries for each file
    return combos.map((combo, idx) => ({
      type: 'story'
      importPath: fileName,
      key: `Combo${idx}`
    }];
  }
}

export default { indexers: [MarkdownIndexer] };

Markdown docs

Consider the following addon that supports pure Markdown docs and uses the headmatter to optionally control the sidebar:

import fs from 'fs/promises';
import path from 'path';

const MarkdownIndexer: Indexer = {
  test: /.md$/,
  createIndex: async (fileName) => {
    const md = await fs.readFile(fileName);
    const { title, name, tags } = parseHeadMatter(md);

    // Sidebar entries for each file
    return [{
      type: 'docs'
      importPath: fileName,
      exportName: 'default',
      title, // optional
      name, // optional
      tags, // optional
      storiesImports: undefined // not needed since we don't ref stories
    }];
  }
}

export default { indexers: [MarkdownIndexer] };

Prior Art

There is already an experimental indexer API in Storybook, it’s just not publicly documented. The primary difference in this proposal:

1. The return type of the current API is a CSFFile object, which is an implementation detail. The new API is a simple interface that doesn’t dictate an implementation.
2. The existing API does not allow for pluggable docs implementations. This proposal adds one.
3. The existing API includes a preset called storyIndexers. The new API is indexers to reflect that it can also produce docs entries.

Deliverables

• Implement the API and expose it as part of the Storybook type system
• Convert the existing indexers to the new API
• Deprecate or remove the old API. Removing would be a breaking change, but since it’s an experimental API we are free to do that.
• Document the API in our addons documentation with examples (as above)
• A release post describing the API, its existing usage (Svelte CSF, MDX stories, pure Markdown docs), and potential future applications

Risks

1. Unsupported use cases. One risk is that there are use cases we should be considering that we're not. We release the API and have to make disruptive breaking changes later to accommodate those use cases.
2. Over-flexible API. Another risk is that the API is somehow too flexible, and supporting it publicly somehow restricts future Storybook architectural decisions.

Unresolved Questions

• Should we think about other sidebar entry types as part of this proposal? External links?
• Should we think about future sidebar UIs, e.g. a “docs” sidebar where each navigation point under a page is a heading in the longform docs?
• Should we consider utilities for writing CSF to make it easier to write loaders/plugins? Our current CSFFile does this, but is not ergonomic and we wouldn’t want to commit to this as a public API.
• Should we deprecate the existing internal API or just release this API as a breaking internal change?

Alternatives considered / Abandoned Ideas

No response

[tobiasdiez]

This sounds like a very good proposal! I also like that it will be possible to index docs and not only stories. As far as I can see, it fulfills all requirements I have on it in the context of native Vue SFC. If you like, I can test early prototypes of the indexer api in this setting.

Concerning the half-sentence, "the addon will provide a vite and webpack plugin to transform those SFCs into CSF at load time" - why is it still necessary to generate CSF? Wouldn't it be better/more flexible/... if the second step would be a loader that returns the story object for a given storyid, without necessarily going through CSF. I understand that this may be more related to #23177, then the Indexer API in itself, but it is one of the main pain points I've encountered while working on the vue addon.

[shilman]

I like this idea @tmeasday. It's probably easier in most cases to make these transformations at runtime than at build time. Let's see what @tobiasdiez say about Vue and maybe @JReinhold @j3rem1e say on the Svelte side. JSON fixture data would also be pretty trivial this way. 🤔

[tobiasdiez]

Yes, this is a nice idea. For the performance, one could try to put as much of the heavy stuff in the build-time loader and only do a bit of light post processing in the transformer.

[JReinhold]

These are all interesting ideas.
Is there something about Node vs. browser we need to consider here? The indexer runs in Node, but I would expect a runtime transformer would need to run in the browser (I might be wrong), which could impose limitations on what's doable and which packages can be used. And how would this work for static builds?

[tmeasday]

Yes, the transformer would run in the browser (both for dev + built storybooks), so the definition would probably have to reference a separate file, like:

const JsonTransformer: Transformer = {
  test: /jsonstories.[tj]sx?$/,
  transformerPath: './transformer.js',
}

[j3rem1e]

I'm not sure to understand everything here, but if the transformer function should run in the browsr, it will not be optimal for Svelte file. The svelte compiler is designed to run on the backend. It is possible to ship it in the browser (like Svelte REPL) but it's slow

[thibaudcolas]

or automatically generate stories from static fixture data (both of which were fairly easy to do with the legacy storiesOf API that CSF replaced)

How about generating stories from dynamic fixture data (say an API call)? I’ve been hoping to do this for a while, it would simplify my integration a lot, but I’ve always put this off due to the uncertainty about storiesOf. Just wondering if there is any room for this in this Indexer API or other ways to add entries to the index "later" in the build process.

edit: I see #23177 does specifically say Fixture data. Some users have fixture data in JSON, YAML, databases, or API servers and want to automatically turn this data into stories.. So I take it that this would still be possible, but only from API calls at build time? (since the Indexer API has async support)

Non-goals

Not a storiesOf replacement. We are not trying to solve storiesOf backwards compatibility. We are betting on static analysis and the solution described here is a “load-time” solution. We may provide a more dynamic solution in the future for dynamic “run-time” generating variants within a story and handling them sensibly across our tools.

I’m concerned to see this called out with this language, as to me it seems #23177 clearly states this Indexer API is intended as a replacement for people currently stuck with storiesOf for other reasons than inertia.

is also served at the URL endpoint /index.json in v7 storybooks.

This is the first time I see this formally described so apologies if this is obvious – is the presence of this index representation and its structure something that’s part of the stable API, that other tools could rely on? Again from my perspective for the purpose of integration with other pattern library tools, it’d be really useful to have control over adding entries to Storybook’s index (what this RFC solves), but also at other points inspecting what stories are defined in Storybook according to its parsing of CSF and Indexer API calls that happened at build time.

[shilman]

How about generating stories from dynamic fixture data (say an API call)?

We generally don't like dynamic fixture data in Storybook since it means that your components are not really isolated. I would recommend considering carefully if that's what you really want to do.

That said, we know people want to do this and Storybook supports it in two different ways.

1. The dynamic stories POC linked from the StoriesOf Deprecation RFC is capable of fetching data and generating stories from it.
2. The Storybook loaders API allows you to asynchronously load data before your story renders.

I’m concerned to see this called out with this language, as to me it seems #23177 clearly states this Indexer API is intended as a replacement for people currently stuck with storiesOf for other reasons than inertia.

This is all worded pretty carefully, but I guess it's not clear enough. The Indexer API is a kind of "meta-replacement". I.e. it's not something you can use directly, but you can use it to build stuff like the dynamic stories POC. We still haven't decided exactly how or whether we want to create a direct replacement for storiesOf, e.g. the Variants ideas mentioned at the end of the StoriesOf Deprecation RFC.

This is the first time I see this formally described so apologies if this is obvious – is the presence of this index representation and its structure something that’s part of the stable API, that other tools could rely on?

Exactly. We already have an indexer API but it's not really documented. The intent of this RFC is to make it public & documented & supported so that people can build tools on top of it.

[tobiasdiez]

One more small remark: while implementing the indexer for the vue stories, I encountered the problem how to calculate the id of a story (based on the story name and some meta info), see tobiasdiez/storybook-vue-addon#38 (comment). If I read the proposal correctly, the indexer doesn't need to do this anymore and this is very good.

[floroz]

Thanks for your work on this proposal.

I am trying to play around with the new Indexer API but I am confused about its usage (I know you are working on providing some documentation soon).

I am using the addon manager API with the new experimental config

// export of the addon for clients to add to their `addons: []` in `.storybook/main.ts`

const vueSFCIndexer: Indexer = {
  test: /\.stories\.vue$/,
  index,
}

export const experimental_indexers: StorybookConfig['experimental_indexers'] = existingIndexers => [
  vueSFCIndexer,
].concat(existingIndexers || [])

The index function is the following:

export async function index(
  fileName: string,
  { makeTitle }: IndexerOptions,
): Promise<IndexInput[]> {
  const code = fs.readFileSync(fileName, { encoding: 'utf-8' }).toString()

  const { meta, stories } = parseSFCStory(code)

  return stories.map(({ title }) => ({
    type: 'story',
    importPath: fileName,
    exportName: title,
    title: makeTitle(meta.title),
  }))
}

If I am understanding correctly the IndexInput type signature

• importPath is going to reference the original filename containing a pre-transformed Story.

Will be up to our custom loader to make sure that Story will be transformed to CSF format (in this case the filename is pointing at *.stories.vue files.
This importPath will be parsed again at story load time where the actual file will be transformed. What is its purpose, since it was not included in the deprecated storyIndexer API?

• exportName is the export name of a single story (can be generated by the individual story title)

• title the title of the meta used to group together in the navigation panel the different Story exportName/name

So for example, if my meta.title is Components/Button, then each exportName can be Primary, Secondary etc.

Is my understanding correct?

[kamranayub]

I just wanted to chime in, and I apologize if this isn't exactly what the Indexer API is meant to support (but it sounds closest to what I want). I haven't done a wide search yet on other issues/discussions.

I am currently hacking Storybook into the Docusaurus lifecycle as a plugin.

You can see it here as a WIP:

excaliburjs/Excalibur@b42edc1

Currently it's not ideal: I have to basically run the Storybook dev server side by side while Docusaurus is running and then embed the dev server URL to iframe stories into our docs. For a static build, it's a little easier, it's mainly dev that is sub-optimal. But in the end, we still are getting a separate "site" that isn't natively integrated with our own.

What I "want" is a "headless" Storybook API so I can create a native Docusaurus integration. That's the long and short of it.

For example, we use Typedoc and it's relatively easy to call their Node API and get a JSON data structure you can use to render your own UI. It's more work in userland but you get a native integration. That's exactly how the Docusaurus Typedoc plugin works and it's awesome.

But AFAIK, before seeing this Indexing RFC, there's no way to call a core API and get all the data we'd need back to render a custom UI. Even the RFC specifies here that we'd need to still go through a builder/loader to get story metadata.

This is the first time I have seen this but I did notice the indexing code when I was trying to figure this out last night, so I will be diving into this more. Perhaps what's there is enough I can piece together something, since the Indexer API gives me the import path which may be just enough + the builder.

I'm happy to move this to a dedicated discussion but this seems like the type of use case this is meant to (at least partially) address.

[psychobolt]

I think currently the storyStoreV7 builder does not take account of the indexer's importPaths. I tried creating my own indexer following the experimental doc guide, using a different importPath from fileName. However, my dynamically generated importPath is not included in the virtual module storybook-stories.js. Another thing to consider is transforming these new CSF files for test-runner

[JReinhold]

Sounds like a bug, can you open an issue with a minimal reproduction?

[psychobolt]

A issue was created by someone else: #25554