Add categories as the top filter, show the most trustful mcps on home page and allow user to browse mcp servers by categories

[ups216]

The current mcp server listing is not easy to navigate and the only way user can do is search by keyword.
User will normally want to look at a certain category, e.g. file system, search, software development tools etc. Another issue is that when MCP servers' number increasing, it's hard for user to determine which server is useful, useable or trustforthy.

So we need to introduce this 'cateogry' system that allow us to use 'category' as top-level filtering to orgainze servers, we want to

• Add a fixed categories list and show this as a menu item (with dropdown) to allow user choose a cateogry and only list the servers in certain category
• Add a page (Listing.tsx) with different filtering labels, for now, we only add "Categories". But we need paging, sorting tools
• On homepage, we need to show Servers in the following orders
  • Offical: which is listed in MCP offical website
  • Search
  • File
  • Dev Tools
  • ... etc. - list them based on commonly intested categories ordering

[ups216]

Solution: "Enhanced Reference Server URL Resolver"

I've improved the mcp_servers_crawler.ts script to correctly process the "Reference Servers" section of the MCP Servers README. The script now properly converts relative URLs to full GitHub repository paths.

Key improvements:

1. Added URL conversion functionality:

   • Created a convertToFullGitHubUrl() function that transforms relative paths like src/brave-search to full GitHub URLs like https://github.com/modelcontextprotocol/servers/tree/main/src/brave-search
   • Added configuration constants for the base GitHub repo URL and default branch

2. Enhanced metadata extraction:

   • Added capability to extract server names and descriptions from Markdown links
   • Created a specialized regex pattern for Reference Servers section to handle their unique format

3. Improved section identification:

   • Enhanced section detection to better handle nested and sub-sections
   • Added support for different Markdown formatting styles

4. Better URL processing:

   • Added helper functions to find and analyze surrounding text for URLs
   • Improved URL normalization for consistency

5. Enhanced output:

   • Added rich metadata to the JSON output file
   • Added improved console logging for better visibility

Example output:

The Reference Server section now correctly shows entries like:

{
  "githubUrl": "https://github.com/modelcontextprotocol/servers/tree/main/src/brave-search",
  "name": "Brave Search",
  "description": "Web and local search using Brave's Search API",
  "type": "Reference Server"
}

How to run:

Use the npm script to run the crawler:

npm run crawl-servers -- --url https://raw.githubusercontent.com/modelcontextprotocol/servers/refs/heads/main/README.md

The script now successfully crawls all 441 servers from the README and stores them correctly in mcp_servers_official_list.json.