ENH: Support source files in any language #1192

speth · 2023-09-09T02:54:37Z

This PR adds support for reading source files in any language supported by Pygments and generating annotated HTML files from them.

I saw the previous work that was attempted on multi-language support (#196/#197), and it's clear that some of the refactoring work that's been done since allowed this feature to be introduced with relatively few changes to the existing code base.

Obviously, this is a fair bit short of being able to execute such examples and collect outputs, but I think it's a useful capability for projects that have multiple language interfaces to be able to present examples this way, even if only the Python ones have all the bells and whistles.

There are a few things that still need to be resolved here, including additional tests and documentation, but I wanted to get some feedback first before going too much further. There is a bit of refactoring of py_source_parser.py that could be done to share some of the implementation with the new BlockParser class, which I'd be happy to do if desired.

For context, I'm one of the developers of Cantera, a library for chemical kinetics that provides interfaces in Python, Matlab, C++, and Fortran. I'm currently working toward a replacement for our home-grown index of examples, which is implemented as a set of extensions to Nikola as we move these examples into our Sphinx-generated site. I think sphinx-gallery provides a great set of features for our Python examples, but I don't want to leave our other interfaces behind. I'm hoping this will also be useful for other projects that include interfaces in multiple languages.

larsoner

Looks like a clean implementation so far! Just some preliminary comments from me. Just for background, how will this help you(r project) specifically? I assume you have some mix of Python and something else, and being able to link to non-executed something-else code will make your doc better...

sphinx_gallery/gen_rst.py

sphinx_gallery/utils.py

speth · 2023-09-12T03:47:07Z

Just for background, how will this help you(r project) specifically? I assume you have some mix of Python and something else, and being able to link to non-executed something-else code will make your doc better...

I updated the PR description to provide some context.

larsoner · 2023-09-14T17:12:51Z

One last testing idea -- can you modify sphinx_gallery/tests/tinybuild to use this functionality? It's really convenient to be able to do make in that dir and look quickly to make sure everything renders properly, plus CircleCI renders it for us (so easy to review by eye). After you add it, you can modify sphinx_gallery/tests/test_full.py -- maybe make a new function that uses the sphinx_app fixture that gives the paths to the source-copied-to-a-tmpdir and rendered site -- to have some lines that make sure that whatever non-Python source file you add gets rendered in at least some way you expect.

speth · 2023-09-19T22:26:15Z

Thanks for the pointers on setting up some integration tests using the tinybuild project. I added a couple of examples there with corresponding tests of the resulting .rst files.

The ci/circleci: build_docs failure:

! LaTeX Error: Unicode character π (U+03C0)
               not set up for use with LaTeX.

Is apparently caused by the fact that I tried to use π in a Julia example. I'm not sure how to fix this other than by not using Unicode in the example.

The failure on sphinx-gallery.sphinx-gallery (Main Linux conda_python310_sphinxDev):

Theme error:
An error happened in rendering the page advanced.
Reason: ImportError("cannot import name '_get_toctree_ancestors' from 'sphinx.environment.adapters.toctree' (/usr/share/miniconda/lib/python3.10/site-packages/sphinx/environment/adapters/toctree.py)")

Doesn't seem to have anything to do with the changes in this PR.

larsoner · 2023-09-19T23:45:02Z

Yes sphinx dev is unrelated so you can ignore it for now. Can you avoid Unicode for now and we can fix it later if needed?

larsoner

Other than a few minor comments, looks good to me!

@lucyleeow will you have time to look? If not I think we can rely on our unit tests in addition to my review here, but this touches some core code so if you can look that's better if possible

larsoner · 2023-09-30T21:18:56Z

doc/configuration.rst

+    Currently, all generated notebooks specify Python as the kernel. After downloading,
+    the user will need to manually change to the correct kernel.


Maybe a follow-up PR? I think in principle changing the skeleton might do it

https://github.com/speth/sphinx-gallery/blob/0157d3f801069f6b805b3f232453d75553e0c6cd/sphinx_gallery/notebook.py#L30

But I have no experience with this so...

Yes, this is on my list for follow-on work.

larsoner · 2023-09-30T22:11:49Z

sphinx_gallery/tests/tinybuild/doc/Makefile

@@ -10,7 +10,7 @@ clean:
 	rm -rf gen_modules/

 html:
-	sphinx-build -b html -d _build/doctrees $(SPHINXOPTS) . _build/html
+	sphinx-build -b html -v -d _build/doctrees $(SPHINXOPTS) . _build/html


I'd rather move the -v to SPHINXOPTS at the top

Sorry, I didn't mean to commit that in the first place.

Change signature of rst_blocks rather than changing contents of file_conf.

Also make parsing of almost all extensions opt-out

for more information, see https://pre-commit.ci

…acter

Allow just '%%' instead of '% %%'. Also, treat the rest of the line as a heading, which is what Matlab does. This is enabled for all languages, as it provides a nice short syntax for just adding headings to the code.

Include non-Python source files in the source code zip, and adjust the file name if it includes non-Python sources. Only generate Jupyter notebook zip file and link if at least some of the files are Jupyter compatible.

lucyleeow · 2023-10-01T01:21:28Z

I'll have a look next week.

lucyleeow

Thanks @speth , this is great and well tested, only some nits.

lucyleeow · 2023-10-02T00:52:07Z

doc/configuration.rst

@@ -211,6 +214,25 @@ consult the `regular expressions`_ module for more details.

        $ sphinx-build -D sphinx_gallery_conf.filename_pattern=plot_specific_example\.py ...

+To parse examples in other languages, add their extensions to ``example_extensions``.


Maybe we could be more explicit in saying this is only parsing and syntax highlighting, not executing (I know its obvious for us)? Something like:

"You can also parse and highlight syntax examples in other languages by adding their extensions to example_extensions, though they will not be executed."

lucyleeow · 2023-10-02T00:52:22Z

doc/configuration.rst

+determined by the file extension. To override Pygments' default file associations, the
+``filetype_parsers`` option can be used to specify a ``dict`` mapping a file extension
+to any of the `pygments language names <https://pygments.org/languages/>`__. For
+example::


If we specify a non default file association, should the extension in example_extensions above be the default one or non-default one? Could we state this ?

I tried to clarify. This option isn't so much about handling the case of an extension with no corresponding lexer (though it can be used that way), but for the case where an extension corresponds to several lexers and Pygments' defaults to the wrong one. For example, Pygments lists '.m as a known extension for Matlab, Objective-C, and Mason, with the default being Objective-C.

lucyleeow · 2023-10-02T00:55:17Z

doc/configuration.rst

+downloads can be enabled using the ``notebook_extensions`` option. For example::
+
+    sphinx_gallery_conf = {
+        "notebook_extensions": {".py", ".jl"}


Again, if a non default extension is used in filetype_parsers, should this be the default or non default extension? Could we state this?

lucyleeow · 2023-10-02T01:05:38Z

doc/syntax.rst

+
+For such examples, the header for the example is defined by the first comment block
+in the file, which must contain a reST title, and may contain any additional reST
+content that should appear above the example. For example, A C++ example could start


Suggested change

content that should appear above the example. For example, A C++ example could start

content that should appear above the first code block. For example, a C++ example could start

lucyleeow · 2023-10-02T01:16:44Z

doc/syntax.rst

+Any text following the special delimiter on the same line will be converted into a reST
+heading (underlined with ``-``). The reST block continues until a line that does not


Does the "(underlined with -)" mean that the header should be underlined with - or it will become a reST header as if it was underlined with e.g., - ?

Also I am not so familiar with other languages, does 'single line' (?) comment languages make supporting the current header syntax not possible? e.g.,

# %% # Header # -----------

?

It means that the underlining will be added, using the - character. I specified this so the user would know what character to use if they wanted other headings at either the same or a different level.

The existing header syntax is also supported. However, in the case of Matlab, which has had the notion of "code cells" for a long time (possibly where this idea originated), the style is that text appearing on the same line as the %% delimiter is treated as a heading, rather than ignored as sphinx-gallery currently does for Python scripts. Beyond Matlab, I thought it was a handy syntax especially in the case where all you want to add is the heading.

Ah yes, sorry I think I got confused.

lucyleeow · 2023-10-02T01:29:33Z

sphinx_gallery/backreferences.py

    thumb, _ = _find_image_ext(
-        os.path.join(target_dir, "images", "thumb", f"sphx_glr_{fname[:-3]}_thumb.png")
+        os.path.join(target_dir, "images", "thumb", f"sphx_glr_{fname.stem}_thumb.png")


This is nice!

lucyleeow · 2023-10-02T01:31:52Z

sphinx_gallery/block_parser.py

@@ -0,0 +1,361 @@
+"""BlockParser divides source files into blocks of code and markup text."""


Suggested change

"""BlockParser divides source files into blocks of code and markup text."""

"""BlockParser divides non `.py` source files into blocks of code and markup text."""

nitpick

lucyleeow · 2023-10-02T02:15:28Z

sphinx_gallery/block_parser.py

+        flag_start = rf"^[\ \t]*(?:{comment_start})\s*"
+
+        self.infile_config_pattern = re.compile(
+            flag_start + r"sphinx_gallery_([A-Za-z0-9_]+)(\s*=\s*(.+))?[\ \t]*\n?",


Nit, r"sphinx_gallery_([A-Za-z0-9_]+)(\s*=\s*(.+))?[\ \t]*\n?" is repeated in py_source_parser.py and is complex enough that we could use a variable? (Even if we refactor in future)

lucyleeow · 2023-10-02T02:52:01Z

sphinx_gallery/block_parser.py

+            ("#= comment =#", "#=", "=#", None),  # Julia multiline
+            ("# comment", "#", None, "#{20,}"),
+            ("// comment", "//", None, "/{20,}"),
+            ("/* comment */", r"/\*", r"\*/", r"/\*{20,}/"),
+            ("% comment", "%", None, "%{20,}"),
+            ("! comment", "!", None, "!{20,}"),
+            ("c     comment", r"^c(?:$|     )", None, None),


Do each of these correspond to a (or more) language? Could we add a comment specifying the associated language?

Yes. I added some examples for each. There are others, but I didn't want to go overboard.

speth · 2023-10-02T16:46:32Z

Thanks for the review, @lucyleeow and @larsoner. I think I've addressed all the review comments.

lucyleeow · 2023-10-02T22:57:32Z

Thank you @speth ! This is a great addition!

… to version 0.15.0 v0.15.0 ------- Support for Python 3.7 dropped in this release. Requirement is now Python >=3.8. Pillow added as a dependency. **Implemented enhancements:** - ENH: Improve logging visibility of errors and filenames `#1225 <https://github.com/sphinx-gallery/sphinx-gallery/pull/1225>`__ (`larsoner <https://github.com/larsoner>`__) - ENH: Improve API usage graph `#1203 <https://github.com/sphinx-gallery/sphinx-gallery/pull/1203>`__ (`larsoner <https://github.com/larsoner>`__) - ENH: Always write sg_execution_times and make DataTable `#1198 <https://github.com/sphinx-gallery/sphinx-gallery/pull/1198>`__ (`larsoner <https://github.com/larsoner>`__) - ENH: Write all computation times `#1197 <https://github.com/sphinx-gallery/sphinx-gallery/pull/1197>`__ (`larsoner <https://github.com/larsoner>`__) - ENH: Support source files in any language `#1192 <https://github.com/sphinx-gallery/sphinx-gallery/pull/1192>`__ (`speth <https://github.com/speth>`__) - FEA Add examples recommender system `#1125 <https://github.com/sphinx-gallery/sphinx-gallery/pull/1125>`__ (`ArturoAmorQ <https://github.com/ArturoAmorQ>`__) (NEWS truncated at 15 lines)

speth force-pushed the multi-src-lang branch 2 times, most recently from 0899482 to 066b1ec Compare September 9, 2023 15:37

speth mentioned this pull request Sep 10, 2023

Reorganize website source material to simplify updating Cantera/enhancements#178

Open

larsoner added the enhancement label Sep 11, 2023

larsoner reviewed Sep 11, 2023

View reviewed changes

speth force-pushed the multi-src-lang branch from f343517 to 2855a9b Compare September 12, 2023 03:45

speth force-pushed the multi-src-lang branch 2 times, most recently from 30c742d to eb4f15e Compare September 19, 2023 22:13

speth force-pushed the multi-src-lang branch 4 times, most recently from ef7e565 to d13963d Compare September 22, 2023 23:53

speth marked this pull request as ready for review September 23, 2023 00:02

speth requested a review from larsoner September 30, 2023 00:43

larsoner reviewed Sep 30, 2023

View reviewed changes

speth added 11 commits September 30, 2023 19:32

Handle example filename extensions besides '.py'

9e95816

Add first version of C++ parser

b1d1ba1

Do syntax highlighting based on input language

6907e7d

Handle ignore blocks and sphinx_gallery_foo flags

2c1420a

Refactor BlockParser base class; add config option to specify parser

89d6dae

Rewrite BlockParser to support any language using pygments

d69193c

Fix some problematic cases in BlockParser

a2da94d

Fix (most) existing tests

907e64a

Change signature of rst_blocks rather than changing contents of file_conf.

Test multilingual parser and fix a few parsing bugs

36b0154

Allow overrider of filename-based language selection

b51711c

Also make parsing of almost all extensions opt-out

Fix a case with bad indentation and add a test

86d3179

speth and others added 21 commits September 30, 2023 19:32

Try to make ruff happy

5e49bc0

Add tests of gallery directives

cc954bf

[pre-commit.ci] auto fixes from pre-commit.com hooks

1efff56

for more information, see https://pre-commit.ci

Add handling for Fortran fixed-format comment style

a0e8715

Make target_file a pathlib.Path

111384e

Preserve backwards-compatible function signatures

f0a88c7

Use explicit list of allowed example extensions

219fd2c

Fix indentation of fully-indented code blocks

3335825

add C++ test to tinybuild

ee223ae

Replace tests of inconsistent block syntax

8837fdf

New improved block parser

cbd2b32

Handle MATLAB docstring convention and add test

9ead453

Fix Julia docstring parsing and add example

ee66b42

Disable notebook downloads by default for non-Python examples

682f4cd

Support rule-style delimiter based on current language's comment char…

eefb3c6

…acter

Add documentation for non-Python parsing options

89ff7d5

Remove debug printing and try to make ruff happy

91d1c1f

Avoid Unicode

3f3af92

Add support for Matlab's native code section delimiter

4ee4d59

Allow just '%%' instead of '% %%'. Also, treat the rest of the line as a heading, which is what Matlab does. This is enabled for all languages, as it provides a nice short syntax for just adding headings to the code.

Test remove_config_comments

8537a94

Handle bulk downloads in various source languages

439eca5

Include non-Python source files in the source code zip, and adjust the file name if it includes non-Python sources. Only generate Jupyter notebook zip file and link if at least some of the files are Jupyter compatible.

speth force-pushed the multi-src-lang branch from 0157d3f to 439eca5 Compare September 30, 2023 23:34

lucyleeow approved these changes Oct 2, 2023

View reviewed changes

Address review comments

2ccfd82

lucyleeow merged commit 846c1a0 into sphinx-gallery:master Oct 2, 2023
17 checks passed

lucyleeow mentioned this pull request Oct 18, 2023

FEA Add examples recommender system #1125

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ENH: Support source files in any language #1192

ENH: Support source files in any language #1192

speth commented Sep 9, 2023 •

edited

larsoner left a comment

speth commented Sep 12, 2023

larsoner commented Sep 14, 2023

speth commented Sep 19, 2023

larsoner commented Sep 19, 2023

larsoner left a comment

larsoner Sep 30, 2023

speth Sep 30, 2023

larsoner Sep 30, 2023

speth Sep 30, 2023

lucyleeow commented Oct 1, 2023

lucyleeow left a comment

lucyleeow Oct 2, 2023

lucyleeow Oct 2, 2023

speth Oct 2, 2023

lucyleeow Oct 2, 2023

lucyleeow Oct 2, 2023

lucyleeow Oct 2, 2023

lucyleeow Oct 2, 2023

speth Oct 2, 2023

lucyleeow Oct 2, 2023

lucyleeow Oct 2, 2023

lucyleeow Oct 2, 2023

lucyleeow Oct 2, 2023

lucyleeow Oct 2, 2023

speth Oct 2, 2023

speth commented Oct 2, 2023

lucyleeow commented Oct 2, 2023

		Currently, all generated notebooks specify Python as the kernel. After downloading,
		the user will need to manually change to the correct kernel.

		@@ -211,6 +214,25 @@ consult the `regular expressions`_ module for more details.

		$ sphinx-build -D sphinx_gallery_conf.filename_pattern=plot_specific_example\.py ...

		To parse examples in other languages, add their extensions to ``example_extensions``.

	content that should appear above the example. For example, A C++ example could start
	content that should appear above the first code block. For example, a C++ example could start

		Any text following the special delimiter on the same line will be converted into a reST
		heading (underlined with ``-``). The reST block continues until a line that does not

		@@ -0,0 +1,361 @@
		"""BlockParser divides source files into blocks of code and markup text."""

	"""BlockParser divides source files into blocks of code and markup text."""
	"""BlockParser divides non `.py` source files into blocks of code and markup text."""

ENH: Support source files in any language #1192

ENH: Support source files in any language #1192

Conversation

speth commented Sep 9, 2023 • edited

larsoner left a comment

Choose a reason for hiding this comment

speth commented Sep 12, 2023

larsoner commented Sep 14, 2023

speth commented Sep 19, 2023

larsoner commented Sep 19, 2023

larsoner left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lucyleeow commented Oct 1, 2023

lucyleeow left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

speth commented Oct 2, 2023

lucyleeow commented Oct 2, 2023

speth commented Sep 9, 2023 •

edited