Better support self hosted Indexing and bridge.

[logan-anderson]

    > 1. What would happen for projects that use Tina's new feature for separate content repositories? 

This is a great question. This was not really taken into consideration when doing our first version of this.

For context, behind the scene TinaCMS uses a "Bridge" to get the content for indexing. For self hosted we use the filesystem bridge by default but for this use case we would need a "Github Bridge" we had one at one point but I think it was removed from the codebase. We can look at adding it back to the codebase for those who want to use a separate content repo.

Although its not available right now I think that this would look something like this.

export default createDatabase({
  bridge: new GithubBridge({owner, ref, repo, token }),
  level: isLocal ? localLevelStore : mongodbLevelStore,
  onPut: isLocal ? localOnPut : githubOnPut,
  onDelete: isLocal ? localOnDelete : githubOnDelete,
});

This would index your content from another source but would still do it only at build time.

What would be the best process to ensure re-indexing occurs in that case?

A)

If you editing from TinaCMS, this should not be an issue because the Datalayer should automatically be updated on save. But if you push commits to the content repo yourself (not through TinaCMS) that could cause an issue. The Datalayer would not know about these changes and would not be updated

Again this was not prioritized, so there won't be any examples of how to do it but I can provide an overview of how we could support this.

I think the could look something like this

Commit in content repo -> Github action to get all changed files in content repo -> Sends updated files to a api route in your main repo.

The api route would look something like this.

import { NextApiHandler } from "next";

import database from "../../.tina/database";

const indexHandler: NextApiHandler = async (req, res) => {
  // use your own auth logic
  const isAuthorized = true;
  if (!isAuthorized) {
    // get the paths from the request body instead of hardcoding them
    const paths: string[] = ["content/home.json"];
    await database.indexContentByPaths(paths);
    return res.status(200).json({ success: true });
  } else {
    res.status(401).json({ success: false });
  }
};

export default indexHandler;

This works right now but unfortunately currently looks at the filesystem which is not super helpful (and would not work for separate content repos)

I made a quick demo video that may help clarify: https://www.loom.com/share/c7d1c4f5bda04ec3a1dacff2a03efac1

B)

Reindexing the entire repo is supported but I was looking at the code and I think there is some cleanup required to make it easier for folks who are self hosting.

I think the API should look like this (not currently supported)

import { NextApiHandler } from "next";

import database from "../../.tina/database";
import config from "../../.tina/config";
import gqlSchema from "../../.tina/__generated__/_schema.json";

const indexHandler: NextApiHandler = async (req, res) => {
  //   use your own auth logic
  const schema = JSON.parse(JSON.stringify(config.schema));
  const result = await database.indexContent({
    tinaSchema: schema,
    gqlSchema,
  });
  console.log(result);
  res.status(200).json({ success: true });
};

export default indexHandler;

Originally posted by @logan-anderson in #3589 (reply in thread)

I think the scope of this work should be to

• Add back the GithubBridge
• Improve the API for reindexing

[coreyaus]

Massive thanks for sharing so much incredibly useful info above @logan-anderson!

I've just created a draft Pull Request that I'll be testing locally to hopefully refine and finalise over the coming days: #3786. I like the approach you outline, and agree it probably makes sense to just identify the paths for all the files that change in the content repository, rather than forcing a full re-index each time.

On the full indexing, is the existing support you mention essentially what's encapsulated in the _indexAllContent function visible in the database/index.ts file (from within the @tinacms/graphql package): https://github.com/tinacms/tinacms/blob/main/packages/@tinacms/graphql/src/database/index.ts#L1026-L1078. Or is there other code somewhere that currently handles the full reindexing process?

In any event, just writing to say thanks for all your helpful thoughts and excellent guidance on this point - hopefully I'll have some luck and everything will work smoothly while testing some of the nuance around this in the next few says 😄

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.