add support for async galleries

[pmeier]

This Pull request fixes #88.

• I completed the issue numbers (#xx) in the sentence above. The word "fixes" should remain in front of each issue
• My PR is tagged as draft when I'm still working on it, and I remove the draft flag when it is ready for review.
• I added one or several changelog.md entries in the appropriate "in progress" section (not the last release one)

b - My PR adds some features:

(Delete this section if not relevant)

• Each issue is well-described, well-labeled and explains/agrees on the expected solution (high-level).
• I added documentation gallery examples showing the new features
• I created tests for each new feature. In particular I have for each feature:
  • nominal tests (several simple ones or a parametrized one, or both)
  • edge tests (valid but very particular cases, for example empty data, daylight savings change day, sparse matrix, etc.)
  • error tests (in which case use with pytest.raises(MyErrorType, match="..."))

[pmeier]

This already works somewhat:

A few issues though that I need to think about how to solve. Since we wrap the whole block in an async function,

1. we need to return the result for it to be displayed, and
2. we need to somehow propagate the locals inside the wrapper function into the globals of the overall script.

[pmeier]

Ok, maybe the issues in #90 (comment) aren't too bad after all:

1. We can use something similar to
   

   mkdocs-gallery/src/mkdocs_gallery/gen_single.py

   Lines 644 to 649 in cc5c4ad

   	# capture output if last line is expression
   	is_last_expr = False
   	if len(code_ast.body) and isinstance(code_ast.body[-1], ast.Expr):
   	is_last_expr = True
   	last_val = code_ast.body.pop().value

   to capture the output that we want to return. The fact that this happens on the AST here makes me believe that we want to perform the patching on the AST level as well.

2. Before returning, we can just simply update script.run_vars.fake_main

   mkdocs-gallery/src/mkdocs_gallery/gen_single.py

   Line 652 in cc5c4ad

   _exec_once(compiler(code_ast, src_file, "exec"), script.run_vars.fake_main)

   with the locals in the function.

[pmeier]

@smarie Could you make a first pass on this? I deviated a little from our plan in #88. Instead of having a feature flag for the async handling, I've opted to detect it, by checking if compiling the code leaves us with an async error. IMO this is robust enough, but I'll let you be the judge of that. The patch will also work with a feature flag, so we can easily switch to that if you prefer.

[pmeier]

Don't worry about the 3.7 and 3.8 CI jobs that are failing. ast.unparse is only available starting 3.9. And that will ultimately removed as explained in #90 (comment).

[smarie]

I was surprised to see this but it is indeed present in sphinx-gallery: https://github.com/sphinx-gallery/sphinx-gallery/blob/master/sphinx_gallery/gen_rst.py#L965

They probably use this as a pre-validator of good syntax. Should we keep it too ? In that case I suggest to keep these original lines and simply add your if _needs_async_handling extra two lines below (dropping _parse_code). That way the change will be easier to spot when comparing with sphinx-gallery code. Of course no hurry with this one, as this is a nitpick

[pmeier]

They probably use this as a pre-validator of good syntax. Should we keep it too ?

That is what we basically have right now. If we find a SyntaxError there, we need to check whether this comes from using async stuff and can't just exit the try branch pre-maturely. In the light of #90 (comment), we could just bubble up a SyntaxError if our async handling doesn't resolve it.

[pmeier]

I've implemented my suggestion in 5e0e636. Please have a look.

[pmeier]

@smarie I'm happy with the functionality. Other than #90 (comment) do you have any other comments that could prevent this from being merged? If no, I'll get on improving the example and documenting this change in general.

As for tests, I see that a lot of them are commented out. How do you want to handle this? Just trust the fact that we can build the examples including the async one?

[smarie]

Thanks @pmeier ! I'll have a more thorough look in the upcoming days, just a quick answer concerning the tests: this project was a port of sphinx-gallery "as close as possible to the original", so that evolutions/improvements/bugfixes could easily be performed on both sides. Obviously this was not ideal (creating a common core lib would do much better), and that forced me to stick with some coding style that I would not have performed from scratch. A side effect of trying to not change much the code design, was that I was totally unable to integrate the original tests - too far from what I would have intuitively done, and obviously very sticky with the design of the internals.

Lacking time, I decided to go for a very crude strategy: the doc page of this project IS the main test. So if you add a gallery example for async, it will be taken into account for coverage for example. That being said, adding a few low-level tests dedicated to your utility functions could make this much more robust to edge cases, and error cases so I encourage you to do so.

I hope this is clear enough, first day of 2024 - still a bit tired :)

[pmeier]

@smarie I've added some text to the example as well as tests. This is ready for another round of review.

[pmeier]

Ping @smarie

[pmeier]

Ping @smarie

[smarie]

All set, thanks a lot for this nice PR @pmeier !

[smarie]

And here is 0.10.0 , showing your great new feature :

https://smarie.github.io/mkdocs-gallery/generated/gallery/plot_12_async/