doc: extensions: add kconfig search extension

[gmarull]

Introduction

This PR adds a new Sphinx extension to handle Kconfig documentation. The way this extension works differs significantly from the current approach, mainly due to technical limitations that will be detailed next. As of today, a bunch of RST pages are generated (one per option, so ~10K) by a script (gen_kconfig_rest.py). These pages get parsed and rendered by Sphinx when the documentation is built. It turns out that adding ~10K individual pages to Sphinx slows things down, to a point we had to create the KCONFIG_TURBO_MODE or html-fast targets. In practice, this means that any viable solution needs to stay away from individual pages. In fact, the way bindings docs work suffer from the same issue, just that we still have less of them.

Technical aspects of the solution proposed here:

• It is a native Sphinx extension (no more CMake hackery required)
• It uses kconfiglib to build a JSON database for all Kconfig symbols
• Introduces a new Sphinx language domain for Kconfig. The domain provides a Sphinx directive (.. kconfig:search::) that can be used to insert a Kconfig search tool on any RST page. When building HTML docs, that directive gets replaced by an actual search form. The :kconfig:option: role is now available to reference options (goes to Kconfig search form pre-selecting the option). Thanks to using a regular Sphinx domain, Kconfig options can also be referenced in other projects using Intersphinx.
• A client-side Javascript library takes care of the rest: fetches the database and, based on user input, it renders Kconfig content dynamically. The library also handles URL hash, that is, if the page holding the Kconfig form is queried with #KCONFIG_OPTION_TO_DISPLAY it will just display it. This allows for permalinks, e.g., CONFIG_SPI.

Pros:

• A native Sphinx extension
• Only a single page is needed: this means docs will be faster by default
• We can quickly search Kconfig options (it's faster than Sphinx search!)

Cons:

• Kconfig options don't have a page now, so search engines will not index them.

Demo

https://teslabs.github.io/zephyr-kconfig-test/kconfig.html

[nashif]

• Kconfig options don't have a page now, so search engines will not index them.

can we just list them somewhere without any individual rst pages for search engines to index with instructions to use the search capability for more details?

[gmarull]

• Kconfig options don't have a page now, so search engines will not index them.

can we just list them somewhere without any individual rst pages for search engines to index with instructions to use the search capability for more details?

Maybe by rendering the whole database will help if crawler is detected: https://developers.google.com/search/docs/advanced/javascript/dynamic-rendering

[mbolivar-nordic]

Maybe by rendering the whole database will help if crawler is detected: https://developers.google.com/search/docs/advanced/javascript/dynamic-rendering

that kind of sounds like an html-slow ;)

[mbolivar-nordic]

It's a cool idea and I played around with it a bit on your demo site. It is indeed fast, which is great. I don't think having a complete list of Kconfig options in our rendered HTML is very useful except for crawlers, so I think that's not a big loss.

One thing I noticed so far is that you chose to narrow down the set of locations from the complete list of locations in the current pages to just one location. This is picking bad results.

For example, look at CONFIG_SPI_2_NRF_SPIM: https://docs.zephyrproject.org/latest/reference/kconfig/CONFIG_SPI_2_NRF_SPIM.html

The search results page is picking the "wrong" location for it:

[gmarull]

It's a cool idea and I played around with it a bit on your demo site. It is indeed fast, which is great. I don't think having a complete list of Kconfig options in our rendered HTML is very useful except for crawlers, so I think that's not a big loss.

One thing I noticed so far is that you chose to narrow down the set of locations from the complete list of locations in the current pages to just one location. This is picking bad results.

For example, look at CONFIG_SPI_2_NRF_SPIM: docs.zephyrproject.org/latest/reference/kconfig/CONFIG_SPI_2_NRF_SPIM.html

The search results page is picking the "wrong" location for it:

Seems to be a bug in the database generation. It's incomplete now, I need to figure out a way to store all the locations and present the information in a better way to avoid things like that:

[mbolivar-nordic]

+1 assuming the issue I pointed out is fixed

[de-nordic]

Would it be possible to add some html param that would allow to capture search list? This would be useful in case when you need to inform somebody on list of Kconfig options that are dedicated to specific subsystem, for example:
https://teslabs.github.io/zephyr-kconfig-test/kconfig.html?seach=CONFIG_FS_LITTLEFS_
would redirect user to page with the CONFIG_FS_LITTLEFS_ already placed in search bar and list narrowed to Kconfig options beginning with CONFIG_FS_LITTLEFS_.

[gmarull]

@mbolivar-nordic @henrikbrixandersen @carlescufi @nashif should be ready now

[henrikbrixandersen]

The documentation changes (and the entire concept) looks fine. Thanks!

I know too little of Sphinx to feel comfortable in reviewing the actual Kconfig extension, though.

canopen changes dropped

[mbolivar-nordic]

I'm giving a +1 for the feature without having reviewed in detail because I'm confident @gmarull will fix any issues that might get reported after this is merged. Thanks for this.

[stephanosio]

@gmarull IIUC, http://docs.zephyrproject.org/latest/reference/kconfig/CONFIG_*.html no longer works with this change; then the Kconfig symbol hint link should be updated as well:

zephyr/scripts/kconfig/kconfig.py

Lines 200 to 205 in f17630b

	SYM_INFO_HINT = """\
	See http://docs.zephyrproject.org/latest/reference/kconfig/CONFIG_{0.name}.html
	and/or look up {0.name} in the menuconfig/guiconfig interface. The Application
	Development Primer, Setting Configuration Values, and Kconfig - Tips and Best
	Practices sections of the manual might be helpful too.\
	"""