Proposal to reduce router development and maintenance costs by using standardized manifest files

[DIYgod]

What feature is it?

To add a new route, developers currently have to create three files in the namespace: maintainer.ts, radar.ts, and router.ts, as well as manually adding documents. This process is cumbersome, non-standardized, error-prone, and challenging for review. To address this issue, a new development and maintenance approach is suggested: simply include a manifest.ts file with all necessary information in a fixed format within the namespace. The RSSHub core will then read this manifest file to load routes. Moreover, automated scripts will generate the required markdown files for documentation purposes.

What problem does this feature solve?

As mentioned above

Additional description

Update2: Implement here #14718

Update1: #14685 (comment)

(Outdated) I propose a preliminary manifest format here:

type Route = {
    name: string;
    url?: string;
    maintainers: string[];
    handler: string;
    example: string;
    parameters?: Record<string, string>;
    description?: string;
    categories: string[];

    features: {
        requireConfig?: string[];
        requirePuppeteer?: boolean;
        antiCrawler?: boolean;
        supportRadar?: boolean;
        supportBT?: boolean;
        supportPodcast?: boolean;
        supportScihub?: boolean;
    };
    radar: {
        source: string[];
        target: string;
    };
};

type Manifest = {
    name: string;
    url?: string;
    categories: string[];
    description?: string;

    routes: Record<string, Route>;
};

For example:

export default {
    name: 'GitHub',
    url: 'https://github.com',
    categories: ['programming'],
    description: `
:::tip
GitHub provides some official RSS feeds:

-   Repo releases: 'https://github.com/:owner/:repo/releases.atom'
-   Repo commits: 'https://github.com/:owner/:repo/commits.atom'
-   User activities: 'https://github.com/:user.atom'
-   Private feed: 'https://github.com/:user.private.atom?token=:secret' (You can find **Subscribe to your news feed** in [dashboard](https://github.com) page after login)
-   Wiki history: 'https://github.com/:owner/:repo/wiki.atom'
:::
    `,

    routes: {
        '/trending/:since/:language/:spoken_language?': {
            name: 'Trending',
            url: 'https://github.com/trending',
            maintainers: ['DIYgod', 'jameschensmith'],
            handler: './trending',
            example: 'https://rsshub.app/github/trending/daily/javascript',
            parameters: {
                since: 'time frame, available in [Trending page](https://github.com/trending/javascript?since=monthly) \'s URL, possible values are: `daily`, `weekly` or `monthly`',
                language: 'the feed language, available in [Trending page](https://github.com/trending/javascript?since=monthly) \'s URL, don\'t filter option is `any`',
                spoken_language: 'natural language, available in [Trending page](https://github.com/trending/javascript?since=monthly) \'s URL',
            },
            description: '',
            features: {
                requireConfig: ['GITHUB_ACCESS_TOKEN'],
                requirePuppeteer: false,
                antiCrawler: false,
                supportRadar: true,
                supportBT: false,
                supportPodcast: false,
                supportScihub: false,
            },

            radar: {
                source: 'github.com/trending',
                target: '/trending/daily',
            },

            // Override the parent's configuration
            categories: ['programming'],
        }
    },
};

This is not a duplicated feature request or new RSS proposal

• I have searched existing issues to ensure this feature has not already been requested and this is not a new RSS proposal.

[SettingDust]

How about manifest.ts or metadata.ts?
Support this since split into the three files is unnecessary.

[DIYgod]

How about manifest.ts or metadata.ts? Support this since split into the three files is unnecessary.

manifest is a good idea, I have updated my content

[NeverBehave]

The reason behind file splits is because we have cold start issue before: every time when there is a need to fetch and compile artifacts based on routes info, or development purposes, we will have to loads all codes even though only small portion of them is useful. It is time consuming to read these files(IO bound) and parse them (as JS file, CPU bound) so I break them as smaller parts based on their usage.

From what I observe so far, at least radar.js and maintainer.js are just objects and should be just fine given parsing should be fast nowadays. At that time I was thinking if performance was still an issue, since maintainer.js is just simple an js object and could be even treated as json file.

Happy to see this change takes place 🥳

[DIYgod]

These three configuration files are significant improvements over previous practices, providing great convenience for current updates. Thanks to @NeverBehave for this innovation.

The most important area that urgently needs improvement now is documents, which have caused the greatest inconvenience and lack of standardization, and are also the biggest obstacle to this proposal implementation.

[SettingDust]

satisfies from TS 5(?) is useful for better code completion

export default {} satisfies RouterManifest

I think it should be in doc and not related to this proposal. Proposed here as an enhancement.

[NeverBehave]

@DIYgod Thanks for the kind word, and the docs part is annoying indeed.

I have some preliminary thoughts of the current proposal:

1. For routes, I think it would be better to use name or path as key as most of the time we probably needs to lookup this array e.g. find maintainers, and if we do want to iterate over them, and also it prevents developer accidentally creates the same path if the list grows longer, easier to identify changes.
2. Multi-language support: it looks like we are embedding markdown into manifest.ts directly, how do we handle en/cn docs generation?

Instead of coupling these markdowns in manifest.ts, I think we could still separate this into different file, say docs.ts

{
  description: {
    "en-us": `some description about this route`
    // if a specific route translate is missing, show corresponded key instead. 
    // However I think there are better way/lib to handle this, just build the right key, not sure if we can enforce this as type check
  },
  "routes.<path>.parameters.since": {
    "en-us": `...`,
    "zh-cn": `...`
  },
}

When building the docs, we can group all of these keys with their prefix, say someroute/docs.ts and we will append folder name as namespace someroute.description and feed to generator, and we just reference them with full path so there is no need to compile each of them separately and then concat them. We can simply pre-populate docs given a list of routes' name and replace all at once.

Still I think multi line markdown in ts is a bit tricky to handle but it should be fine...?

---

The following might be a bit nitpicks :P so just leave here for some ideas.

1. Since we are also includes maintainers, maybe it is a good time to switch to handles. An example would be https://github.com/NixOS/nixpkgs/blob/master/maintainers/maintainer-list.nix, where each maintainer register themselves into a central source of truth and use handles to represent themselves, so if they changed their username or switch account we could handle these case better. We could also grow other attributes like teams. However for now we could just use github username as default handle name would be username most of the time.
2. For options like requirePuppeteer, antiCrawler, etc. I think it would be better to put them under options. It might be helpful for type checks as we can create subtype(?)
3. Instead of having a top level categories: ['programming'],(to me it is acting as defaultCategories?), maybe we should calculate this attributes based on routes? If we provide override options. Not entirely sure how this would work -- if this is used to place routes to the correct category, it has to be under exactly one(1) categories?

[SettingDust]

Instead of coupling these markdowns in manifest.ts, I think we could still separate this into different file, say docs.ts

I think if the doc is split into a file. tsx/mdx is better than the js object. Just render them with the manifest/metadata

[DIYgod]

@NeverBehave

1. Good idea
2. I am worried that multi-language will increase complexity. I want to keep it as simple as possible. My ideal state is for multi-language websites to use English, and single-language websites to use English or the corresponding language. In addition, using translation service instead of manual maintenance is also a better method. We can only retain one language, and other languages can be automatically translated.
3. I want to strongly bind the document with the route. If we separate a docs.ts, I am worried that there will be inconsistencies with the route during development. And I want the markdown file to be completely generated by scripts rather than need manually pre-populate. Indeed, I also found that it is not convenient to write markdown in ts files, but I don't have a better idea.
4. Same as above, I want to keep it as simple as possible. I think the GitHub handle is sufficient for current use.
5. Good idea
6. Yes, if the route does not define a category, use the parent's category. The calculation is still based on the route. It can be multiple because some routes simultaneously meet multiple categories, such as "Bilibili Anime" can be both Multimedia and ACG. When rendering, it will be placed in two documents.

[NeverBehave]

I understand the fact that we want to keep it simple as we goes, while from my pov, if we strip out this flexibility that allow maintainer/community to add custom notes specific to their language it would a great setback.

With that being said, there are two things to add here:

• a language attr 'lang' so we only compile route to correspond language. I am not entirely sure how's the reading experience if we go with mixed language docs, direct mixing probably not what we want
• We could also use 'manifest.tsx, a separate file to keep all markdown/multiline text attributes, and merge them to a single object when processing it?

[DIYgod]

I think it's a good idea to use separate files for different languages, because the name and url may vary depending on the language. In this way, we can have multiple manifest files manifest.[lang].ts, similar to subtitle files.

[songkeys]

If we just export some customized "middleare" functions to define manifest metadata and write handler logic at the same time so that we don't need many manifest files, and also ensure the typings easily.

Example:

// index.ts
import { defineRootRoute } from "common"
import trending from "./trending"

export default defineRootRoute({
  name: "GitHub",
  url: "https://github.com/",
  description: "...",
  // ... other basic metadata,
  routes: [
    trending
  ]
})

// trending.ts
import { defineSubRoute } from "common"

export default defineSubRoute({
  name: 'Trending',
  description: "...",
  url: "/trending/:since/:language/:spoken_language?"
  // ... other basic metadata,
  route: "./trending",
  parameters: {
    since: {
      type: "string",
      description: "time frame, available in [Trending page](https://github.com/trending/javascript?since=monthly) \'s URL, possible values are: `daily`, `weekly` or `monthly`",
    },
    // ... we could even use "zod" to define the parameters for more strict typings
  },
  handler: async (ctx) => {
     // ...
  }
})