Indexing of Docs only able to grab entrypoint page

[JarradJarrad]

1. If you can, please include a screenshot of your problem

When attempting to index DSPy docs via the chat window, I get this:

When attempting to index the same docs via cursor settings -> features -> docs section, I get this:

I have tried all of the below with the github repo the docs are built from, same thing happens.

2. Please include the name of your operating system

Version: 0.33.4
VSCode Version: 1.86.2
Commit: 2dd5b2855ac68f3a3d3babee87d22b399a9db7d0
Date: 2024-05-11T02:05:08.197Z
Electron: 27.2.3
ElectronBuildId: undefined
Chromium: 118.0.5993.159
Node.js: 18.17.1
V8: 11.8.172.18-electron.0
OS: Darwin arm64 23.1.0

3. If you can, steps to reproduce are super helpful

1. navigate to cursor settings -> features -> docs section

2. click 'add new doc'

3. enter the following (with or without /docs/ and with or without trailing slashes, it all results in the same outcome)
   

4. wait while Cursor indexes the docs

5. all it indexes is actually just the entrypoint page