-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Regular expressions removed from sphinx/domains/c.py (e.g. c_funcptr_sig_re) break downstreams #7421
Comments
I may have missed it, but do not believe that functionality was documented before, and it was thus an implementation detail. |
Hm, it seems I have to remove This seems to work for the time being. Not entirely sure whether the output will be the same now though. |
Right, I may have been a bit to fast in my scanning of the file. Note again that all of this is playing with internals of Sphinx so not too many guarantees can be made. The duplicate description handling may not be needed any more, or at least it may hide some underlying problems in the documentation elsewhere. I suggest starting out without this. The handling of function-like macros is redundant and can simply be removed. The Name
====
CEC_ADAP_G_CAPS - Query device capabilities
Synopsis
========
.. c:function:: int ioctl( int fd, CEC_ADAP_G_CAPS, struct cec_caps *argp )
:name: CEC_ADAP_G_CAPS
Arguments
=========
``fd``
File descriptor returned by :c:func:`open() <cec-open>`.
``argp`` Changing it to the following may be a way forward Name
====
.. c:macro:: CEC_ADAP_G_CAPS
Query device capabilities
Synopsis
========
:c:expr:`ioctl(fd, CEC_ADAP_G_CAPS, argp)
Arguments
=========
:c:expr:`int` ``fd``
File descriptor returned by :c:func:`open() <cec-open>`.
:c:expr:`struct cec_caps *` ``argp`` (Except: Note that
Name
====
.. c:macro:: CEC_ADAP_G_CAPS
Query device capabilities
Synopsis
========
.. c:namespace:: @desc_CEC_ADAP_G_CAPS
:c:expr:`ioctl(fd, CEC_ADAP_G_CAPS, argp)
Arguments
=========
.. c:var:: int fd
File descriptor returned by :c:func:`open() <cec-open>`.
.. c:var:: struct cec_caps *argp
See the Sphinx documentation for the C++ domains |
The namespacing directives are in the 3.x branch (future v3.1) so I'll close this for now. Feel free to reopen, or open other issues, if you encounter problems. |
FYI the import error exists since commit 0f49e30#r38750737 |
…e30c) Extension error: Could not import extension linuxdoc.cdomain .. \ (exception: cannot import name 'c_funcptr_sig_re' from .. Issue: `sphinx-doc/sphinx#7421`_ .. _0f49e30c: sphinx-doc/sphinx@0f49e30#r38750737 .. _sphinx-doc/sphinx#7421: sphinx-doc/sphinx#7421 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
References: bsc#1166965 From 3 on the internal API that the build system uses was rewritten in an incompatible way. See sphinx-doc/sphinx#7421 and https://bugzilla.suse.com/show_bug.cgi?id=1166965#c16 for some details.
Hey, any updates on this? I get an error when running 'make htmldocs' for linux 5.7 `$ make htmldocs Extension error: |
Well, not other than 3.1 has been released, so the namespacing trick shown above can be used. Otherwise I believe there is nothing left to do on the Sphinx side. If new features are desired, feel free to open issues. |
@dwlsalmeida did you follow kernel-docs and running make in a virtualenv? If you have problems with the kernel build, please first read the kernel-docs and scrap or contact the linux-doc mailing list. update 2020-08-12: linux-doc ML "Documentation: build failure with sphinx >= 3.0.0" |
@jakobandersen @return42 Hey thanks for chiming in! It works after running make in a virtualenv. Thanks. |
…import name 'c_funcptr_sig_re' from 'sphinx.domains.c' Hi, On Sun, Aug 09, 2020 at 03:23:27PM +0200, Salvatore Bonaccorso wrote: > Hi > On Thu, Apr 09, 2020 at 05:04:54AM +0530, Bhaskar Chowdhury wrote: > > On 13:25 Wed 08 Apr 2020, Jonathan Corbet wrote: > > > On Wed, 8 Apr 2020 17:07:05 +0530 > > > Bhaskar Chowdhury <unixbhaskar@gmail.com> wrote: > > > > > > > Extension error: > > > > Could not import extension cdomain (exception: cannot import name > > > > 'c_funcptr_sig_re' from 'sphinx.domains.c' > > > > (/usr/lib/python3.8/site-packages/sphinx/domains/c.py)) > > > > Apr 08 16:48:46 enabling CJK for LaTeX builder > > > > Apr 08 16:48:46 CC kernel/power/poweroff.o > > > > make[1]: *** [Documentation/Makefile:81: htmldocs] Error 2 > > > > make: *** [Makefile:1549: htmldocs] Error 2 > > > > make: *** Waiting for unfinished jobs.... > > > > > > This is weird, to say the least. But I think the "python3.8" in the > > > message above says everything you need to know. If you're running with > > > an unreleased version of Python, it's not entirely surprising that you > > > might run into trouble with a complex package. > > > > > > jon > > > > Thank you Jon...will investigate more and keep your suggestion in mind. > > The problem is actually related to changes happening in Sphinx 3.0.0. > There is the followign issue filled upstream: > > sphinx-doc/sphinx#7421 > > 'c_funcptr_sig_re' was removed upstream in sphinx v3.0.0b1 and so the > kernel documentation build fails. This is reproducible with a recent > sphinx version (in attached case it is 3.2.0): > > $ make PYTHON=python3 xmldocs > SPHINX xmldocs --> file:///home/build/linux/Documentation/output/xml > make[2]: Nothing to be done for 'xml'. > Running Sphinx v3.2.0 > > Extension error: > Could not import extension cdomain (exception: cannot import name 'c_funcptr_sig_re' from 'sphinx.domains.c' (/usr/lib/python3/dist-packages/sphinx/domains/c.py)) > make[1]: *** [Documentation/Makefile:115: xmldocs] Error 2 > make: *** [Makefile:1655: xmldocs] Error 2 > > Distribution reports related to this issue: > https://bugs.debian.org/963636 > https://bugs.archlinux.org/task/66178 > https://bugs.archlinux.org/task/66156 As a workaround to make the documentation build again (but known that parts of the documentation will be broken), we could drop the cdomain extension. Regards, Salvatore From 9e81028c7e5f8f33f67a768b9c306536fd8e688f Mon Sep 17 00:00:00 2001 From: Salvatore Bonaccorso <carnil@debian.org> Date: Wed, 12 Aug 2020 09:11:51 +0200 Subject: [PATCH] Documentation: Drop use of 'cdomain' sphinx extension module 'c_funcptr_sig_re' was removed upstream in sphinx v3.0.0b1 and so the kernel documentation build fails sphinx 3.x. As a (temporary) workaround to make the documentation build, the 'cdomain' sphinx extension can be removed from the Documentation build configuration file, although some parts using the feature will be broken. sphinx-doc/sphinx#7421 describes the workaround. Cc: Markus Heiser <markus.heiser@darmarit.de> Cc: Justin M. Forbes <jforbes@fedoraproject.org> Link: https://lore.kernel.org/linux-doc/20200809132327.GA145573@eldamar.local/ Link: https://bugs.archlinux.org/task/66178 Link: https://bugs.archlinux.org/task/66156 Link: https://bugs.debian.org/963636 Signed-off-by: Salvatore Bonaccorso <carnil@debian.org>
@jakobandersen I'm the maintainer of the media subsystem at the Linux Kernel. Just to put some context, when dealing with the Linux Kernel documentation, we need to document system calls. The most important system calls are implemented via libc, usually with an userspace with the same name (or with a close name) as the actual function call. As the goal of such documentation is to document how an userspace program will do a system call, we ended referring to the libc name for it. So, an userspace program opening a device node under So, while it could make sense to nave a global Kernel documentation for some system calls like Those subsystem-specific system call documentation may refer to a global system call definition at the at the main namespace (although currently, there's no such cross-references ATM). In the specific case of However, taking the HDMI Consumer Electronics subsystem Control (CEC) as an example, all files under the Didn't work. I'm still getting those warnings, as the documentaion of
In time: I'm building it with Sphinx 3.2.1. |
Thanks for the extra context. Feel free to bring more examples so we can find a good solution. (Do you have a link to where I can get the code with the example?) |
Just saw the question before the patch: yes, the namespacing is per rst file. |
I placed the code here: https://git.linuxtv.org/mchehab/experimental.git/log/?h=test_cec_sphinx3 You can build the docs with:
But this will take a lot of time to build, and will generate about 3K warnings due to the lack of the cdomain (Btw, it sounds that building the docs with Sphinx 3.x takes twice the time as it used to take with Sphinx 2.x). In order to get a cleaner build, I'm doing, instead:
|
Thanks. There was a bug in 3.x, being fixed for 3.2.1 that had a massive impact on the run time. Feel free to open an issue if there are still performance problems. |
Em Wed, 23 Sep 2020 13:53:18 +0200 Mauro Carvalho Chehab <mchehab+huawei@kernel.org> escreveu: > Em Wed, 23 Sep 2020 11:01:06 +0200 > Mauro Carvalho Chehab <mchehab@kernel.org> escreveu: > > > Em Tue, 22 Sep 2020 10:56:56 -0600 > > Jonathan Corbet <corbet@lwn.net> escreveu: > > > > > On Tue, 22 Sep 2020 10:23:46 +0300 > > > Jani Nikula <jani.nikula@linux.intel.com> wrote: > > > > > > > > Should we be converting the kernel-doc script to output 3.0-style markup > > > > > for structs? > > > > > > > > IIUC it's not backwards compatible with Sphinx pre-3.0, so you'd either > > > > have to require Sphinx 3.0 or have kernel-doc emit both, depending on > > > > the Sphinx version. > > > > > > Certainly before requiring 3.0 we should, like, actually make it work with > > > 3.0... People with time to do that appear to be rather scarce at the > > > moment, though. I'll get to it eventually but I have a few other things > > > on my plate as well. > > > > Tried to build now with Sphinx 3.2.1. There are thousands of warnings: > > > > build succeeded, 3423 warnings. > > > > Like those: > > > > Documentation/driver-api/basics.rst: WARNING: Duplicate C declaration, also defined in 'core-api/workqueue'. > > Declaration is 'unsigned long delay'. > > > > And those: > > > > ./include/uapi/linux/firewire-cdev.h:38: WARNING: Type must be either just a name or a typedef-like declaration. > > If just a name: > > Error in declarator or parameters > > Invalid C declaration: Expected identifier in nested name, got keyword: struct [error at 6] > > struct fw_cdev_event_iso_interrupt > > ------^ > > If typedef-like declaration: > > Error in declarator or parameters > > Invalid C declaration: Expected identifier in nested name. [error at 34] > > struct fw_cdev_event_iso_interrupt > > ----------------------------------^ > > > > If I'm not mistaken, one of the reasons why cdomain was added is > > due to the duplicate C declaration. > > > > - > > > > I guess changing kernel-doc to produce Sphinx 3.x syntax and > > raise the bar requiring 3.0 to be the minimal version would be > > one alternative, but this won't solve the duplicated C declarations. > > > > Does Sphinx 3.x have some way to enable "local" C references? > > > > Another one could be to make cdomain compatible with 3.x and > > make it handle the differences. Not sure how easy/hard would be > > to do that. > > Ok, I checked the code of cdomain.py. It sounds that Markus > tried to make cdomain compatible with Sphinx 3.x[1]: > > return42/linuxdoc@48f09de > > [1] Yet, he ended disabling cdomain.py at his latest commit there: > > return42/linuxdoc@70673dc#diff-40ccfb63661f88fd819794047febd54d > > (Basically, if Sphinx > 3.0, the cdomain.py is just an alias for c.py) > > I was hoping that porting such change would be enough for docs to build > with Sphinx 3.2, but it isn't[2]. It seems that the entire Cdomain code > was re-written: > > sphinx-doc/sphinx@0f49e30#r38750737 > > There are now some environment vars that are required. Some of them > have things like: > > lastSymbol = self.env.temp_data['c:last_symbol'] # type: Symbol > assert lastSymbol > > If those aren't found, Sphinx crashes. > > So, making anything that would be compatible with both Sphinx 2.x > and Sphinx 3.x seems very painful. > > It sounds to me that we need to make a choice here about how to > proceed. We have a few alternatives: > > 1) stick forever with Sphinx 2.x syntax. We may end needing to > support a fork of Sphinx, if we follow this direction, as > nobody currently fix bugs on older versions of Sphinx; > > 2) switch to use Sphinx 3.x c domains and drop support for > version < 3.x; Based on what it was described here: sphinx-doc/sphinx#7421 I worked on a RFC patch converting one of the media files (the CEC one, mentioned at the issue) in order to use the new Sphinx 3.1 C domain. Yet, it sounds that the c:namespace is broken on Sphinx 3.2.1: Documentation/userspace-api/media/cec/cec-func-close.rst:23: WARNING: Duplicate C declaration, also defined in 'media/dvb/video-fclose'. Declaration is 'int close( int fd )'. Documentation/userspace-api/media/cec/cec-func-close.rst: WARNING: Duplicate C declaration, also defined in 'media/dvb/video-fclose'. Declaration is 'int close(int fd)'. Documentation/userspace-api/media/cec/cec-func-close.rst: WARNING: Duplicate C declaration, also defined in 'media/dvb/video-fclose'. Declaration is 'int fd'. Documentation/userspace-api/media/cec/cec-func-open.rst:22: WARNING: Duplicate C declaration, also defined in 'media/dvb/video-fopen'. Declaration is 'int open( const char *device_name, int flags )'. Documentation/userspace-api/media/cec/cec-func-open.rst: WARNING: Duplicate C declaration, also defined in 'media/dvb/video-fopen'. Declaration is 'int open(const char *deviceName, int flags)'. Documentation/userspace-api/media/cec/cec-func-open.rst: WARNING: Duplicate C declaration, also defined in 'media/dvb/video-fopen'. Or maybe it has to be defined on each individual .rst file that uses an specific domain? Thanks, Mauro RFC: convert CEC userspace docs to use Sphinx 3.1 domains Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
It would be a lot better if the namespace can be defined together with the file containing the TOC.
Every time a syscall is referred there, it is for the V4L subsystem. The same pattern applies to other subsystems. Btw, we have files using an extension that converts plain C files into markups (called kernel-doc). If the file that includes them would belong to a c namespace, all such markups should also be considered to belong to the same domain by default. Another thing: how can the namespace be overrided on a reference? I mean, if we have two references (like the c:function::open() that could be declared on the virtual filesystem namespace and also at the V4L2 namespace), how can a single rst file can reference to both? |
This is unfortunately not something that can be addressed in the C domain alone. Each rst file is parsed into doctrees individually, and only in a later build phase are cross-references resolved, along with a lot of other transformations.
Note that the namespacing affects all declarations, also for structs. Assuming that is fine, my best recommendation is somehow to get an appropriate namespace directive inserted in the top of the relevant files. There are configuration variables in Sphinx for inserting extra rst (https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-rst_prolog), but they are applied to all documents.
You would need to qualify the name with a leading dot to make the lookup start at global scope. E.g., .. c:function:: int open(const char *path)
.. c:namespace:: @cec
.. c:function:: int open(const char *path)
.. c:namespace:: @cec.@foo
.. c:function:: int open(const char *path)
- The root: :c:func:`.open`
- In cec: :c:func:`.@cec.open` or :c:func:`@cec.open`
- In cec.foo: :c:func:`.@cec.@foo.open` or :c:func:`open`
Or with prettier names:
- The root: :c:func:`open <.open>`
- In cec: :c:func:`open <@cec.open>`
- In cec.foo: :c:func:`open <@foo.open>` |
Ok. This will make harder to maintain the Kernel documentation, but let's try to live with that. Btw, I just opened an issue related to the Kernel documentation build: Building the docs with Sphinx 3.x are generating 1725 warnings. While some of them are due to namespace conflicts, others seem to be caused by some issues at the new Sphinx C domain code. |
…sphinx-doc/sphinx@0f49e30c) Extension error: Could not import extension linuxdoc.cdomain .. \ (exception: cannot import name 'c_funcptr_sig_re' from .. Issue: `sphinx-doc/sphinx#7421`_ .. _0f49e30c: sphinx-doc/sphinx@0f49e30#r38750737 .. _sphinx-doc/sphinx#7421: sphinx-doc/sphinx#7421 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de> Link: return42/linuxdoc@48f09de Signed-off-by: Seth Forshee <seth.forshee@canonical.com>
'c_funcptr_sig_re' was removed upstream in sphinx v3.0.0b1. This causes "make htmldocs" to fail on any distro (such as Arch Linux) that uses Sphinx 3.x. Without this patch, an error like this occurs: Could not import extension cdomain (exception: cannot import name 'c_funcptr_sig_re' from 'sphinx.domains.c' (/usr/lib/python3.8/site-packages/sphinx/domains/c.py)) This problem was discussed in [1]. The Sphinx problem that caused all this is documented in github[2], and Markus wrote this patch (with slightly different wording in the commit log) and posted it to github [3]. (New wording by jhubbard, but otherwise the patch is unchanged.) [1] https://lore.kernel.org/lkml/579fbb36-515d-5af4-f7dc-c092d29fddd8@darmarit.de/ [2] sphinx-doc/sphinx#7421 [3] return42/linuxdoc@48f09de2 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de> Cc: Justin M. Forbes <jforbes@fedoraproject.org> Cc: Salvatore Bonaccorso <carnil@debian.org>
The older versions will need a sphinx fix to work with Sphinx >= 3. See: sphinx-doc/sphinx#7421 Closes: https://bugs.gentoo.org/771597 Package-Manager: Portage-3.0.13, Repoman-3.0.2 Signed-off-by: Mike Pagano <mpagano@gentoo.org>
…sphinx-doc/sphinx@0f49e30c) Extension error: Could not import extension linuxdoc.cdomain .. \ (exception: cannot import name 'c_funcptr_sig_re' from .. Issue: `sphinx-doc/sphinx#7421`_ .. _0f49e30c: sphinx-doc/sphinx@0f49e30#r38750737 .. _sphinx-doc/sphinx#7421: sphinx-doc/sphinx#7421 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de> Link: return42/linuxdoc@48f09de Signed-off-by: Seth Forshee <seth.forshee@canonical.com> Signed-off-by: Ondrej Kubik <ondrej.kubik@canonical.com>
Describe the bug
c_funcptr_sig_re
amongst other regular expressions have been removed from sphinx/domains/c.py in commit 0f49e30. These are used in the Linux kernel html documentation scripts.When trying to compile the HTML documentation for Linux using sphinx 3.0.0 this leads to:
This change was introduced without any notification in the changelog of this release and breaks downstream builds of the Linux Kernel in distributions (when building the HTML documentation).
To Reproduce
Steps to reproduce the behavior:
Expected behavior
The Linux HTML documentation is generated.
Your project
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/sphinx/cdomain.py#n40
Screenshots
If applicable, add screenshots to help explain your problem.
Environment info
Additional context
Add any other context about the problem here.
The text was updated successfully, but these errors were encountered: