diff --git a/_doc/sphinxdoc/source/blog/2015/2015-10-10_tips_sphinx.rst b/_doc/sphinxdoc/source/blog/2015/2015-10-10_tips_sphinx.rst index 37998165a..89192d157 100644 --- a/_doc/sphinxdoc/source/blog/2015/2015-10-10_tips_sphinx.rst +++ b/_doc/sphinxdoc/source/blog/2015/2015-10-10_tips_sphinx.rst @@ -30,12 +30,12 @@ **latex formulas** - The page `Math support in Sphinx `_ + The page `Math support in Sphinx `_ explains how to set up math environment (latex or mathjax). But if you add matplotlib to convert equations into images (`matplotlib.sphinxext.mathmpl `_), the sphinx extension - `sphinx.ext.pngmath is disabled `_. + `sphinx.ext.pngmath is disabled `_. Matplotlib extension has some limitations. ``\text`` does not work. **formulas in docstring** diff --git a/_doc/sphinxdoc/source/blog/2015/2015-12-12_sphinx_extensions.rst b/_doc/sphinxdoc/source/blog/2015/2015-12-12_sphinx_extensions.rst index eb20050b4..d1e1db083 100644 --- a/_doc/sphinxdoc/source/blog/2015/2015-12-12_sphinx_extensions.rst +++ b/_doc/sphinxdoc/source/blog/2015/2015-12-12_sphinx_extensions.rst @@ -6,7 +6,7 @@ :categories: sphinx The following repository - `birkenfeld/sphinx-contrib/ `_ + `birkenfeld/sphinx-contrib/ `_ contains many useful extensions to improve the rendering of `Sphinx `_ documentation: diff --git a/_doc/sphinxdoc/source/blog/2017/2017-05-17_sphinx.rst b/_doc/sphinxdoc/source/blog/2017/2017-05-17_sphinx.rst index 84a629b14..879942465 100644 --- a/_doc/sphinxdoc/source/blog/2017/2017-05-17_sphinx.rst +++ b/_doc/sphinxdoc/source/blog/2017/2017-05-17_sphinx.rst @@ -5,7 +5,7 @@ :date: 2017-05-17 :categories: documentation - `Sphinx `_ + `Sphinx `_ has released a new version 1.6.1 but it breaks a couple of packages. `sphinx_gallery `_ diff --git a/_doc/sphinxdoc/source/blog/2017/2017-05-21_sphinx.rst b/_doc/sphinxdoc/source/blog/2017/2017-05-21_sphinx.rst index 830602c48..e7470ad9f 100644 --- a/_doc/sphinxdoc/source/blog/2017/2017-05-21_sphinx.rst +++ b/_doc/sphinxdoc/source/blog/2017/2017-05-21_sphinx.rst @@ -5,7 +5,7 @@ :date: 2017-05-21 :categories: documentation - `Sphinx `_ + `Sphinx `_ has released a new version 1.6.1. Other packages updated their code but some issues remain. I had to modify the code of the file diff --git a/_doc/sphinxdoc/source/blog/2017/2017-08-25_sphinxgallery.rst b/_doc/sphinxdoc/source/blog/2017/2017-08-25_sphinxgallery.rst index b2e69fb24..cdb352473 100644 --- a/_doc/sphinxdoc/source/blog/2017/2017-08-25_sphinxgallery.rst +++ b/_doc/sphinxdoc/source/blog/2017/2017-08-25_sphinxgallery.rst @@ -12,7 +12,7 @@ There is a discussion about it on `scikit-learn/9189 `_ and the package is now included in the - `sources `_. + `sources `_. It is not really bothering for a small package like this. But still, it is extra work. I realize open source holds on some kind of magic diff --git a/_doc/sphinxdoc/source/contribute.rst b/_doc/sphinxdoc/source/contribute.rst index f37178187..ca428a26c 100644 --- a/_doc/sphinxdoc/source/contribute.rst +++ b/_doc/sphinxdoc/source/contribute.rst @@ -282,7 +282,8 @@ It relies on epkg:`pyquickhelper`. Generation ++++++++++ -The documentation can be written using `RST `_ format +The documentation can be written using `RST +`_ format or `javadoc `_ format. The documentation can generated by: diff --git a/_doc/sphinxdoc/source/glossary.rst b/_doc/sphinxdoc/source/glossary.rst index acd432457..ba0f9beef 100644 --- a/_doc/sphinxdoc/source/glossary.rst +++ b/_doc/sphinxdoc/source/glossary.rst @@ -11,6 +11,6 @@ Glossary RST or ReSTructured Text This format is used to write the documentation of most of Python module. - The module `Sphinx `_ compiles it into + The module `Sphinx `_ compiles it into a HTML pages. See `RST syntax `_ or `RST cheat sheet `_. diff --git a/_doc/sphinxdoc/source/tutorial/sphinx.rst b/_doc/sphinxdoc/source/tutorial/sphinx.rst index fd6cf7b46..1ebc69b25 100644 --- a/_doc/sphinxdoc/source/tutorial/sphinx.rst +++ b/_doc/sphinxdoc/source/tutorial/sphinx.rst @@ -20,7 +20,7 @@ Simple extensions ----------------- :epkg:`Sphinx` implements many -`markups `_. +`markups `_. This module adds a couple of them. Many cheat sheets (see `cheat sheet 1 `_, `cheat sheet 2 `_, @@ -584,7 +584,7 @@ does and a little bit more: This design is described by an empty module: -* `documentation `_ +* `documentation `_ * `github/python3_module_template `_ Blog Post diff --git a/_unittests/ut_cli/data/glossary.rst b/_unittests/ut_cli/data/glossary.rst index acd432457..ba0f9beef 100644 --- a/_unittests/ut_cli/data/glossary.rst +++ b/_unittests/ut_cli/data/glossary.rst @@ -11,6 +11,6 @@ Glossary RST or ReSTructured Text This format is used to write the documentation of most of Python module. - The module `Sphinx `_ compiles it into + The module `Sphinx `_ compiles it into a HTML pages. See `RST syntax `_ or `RST cheat sheet `_. diff --git a/_unittests/ut_cli/data/td1a_unit_test_ci.ipynb b/_unittests/ut_cli/data/td1a_unit_test_ci.ipynb index 4b4f15b58..ee616bb90 100644 --- a/_unittests/ut_cli/data/td1a_unit_test_ci.ipynb +++ b/_unittests/ut_cli/data/td1a_unit_test_ci.ipynb @@ -438,7 +438,7 @@ "source": [ "## Ecrire la documentation\n", "\n", - "L'outil est le plus utilis\u00e9 est [sphinx](http://www.sphinx-doc.org/en/stable/). Saurez-vous l'utiliser ?" + "L'outil est le plus utilis\u00e9 est [sphinx](https://www.sphinx-doc.org/en/master/). Saurez-vous l'utiliser ?" ] }, { diff --git a/_unittests/ut_helpgen/data_project/pp/_doc/sphinxdoc/source/index.rst b/_unittests/ut_helpgen/data_project/pp/_doc/sphinxdoc/source/index.rst index a7a4deb76..ea524d85a 100644 --- a/_unittests/ut_helpgen/data_project/pp/_doc/sphinxdoc/source/index.rst +++ b/_unittests/ut_helpgen/data_project/pp/_doc/sphinxdoc/source/index.rst @@ -45,7 +45,7 @@ python3_module_template :alt: size **Links:** `github `_, -`documentation `_, +`documentation `_, `travis `_, :ref:`l-README`, :ref:`blog `, diff --git a/_unittests/ut_helpgen/notebooks_svg/td1a_unit_test_ci.ipynb b/_unittests/ut_helpgen/notebooks_svg/td1a_unit_test_ci.ipynb index 83dff63a7..36e29647b 100644 --- a/_unittests/ut_helpgen/notebooks_svg/td1a_unit_test_ci.ipynb +++ b/_unittests/ut_helpgen/notebooks_svg/td1a_unit_test_ci.ipynb @@ -437,7 +437,7 @@ "source": [ "## Ecrire la documentation\n", "\n", - "L'outil est le plus utilis\u00e9 est [sphinx](http://www.sphinx-doc.org/en/stable/). Saurez-vous l'utiliser ?" + "L'outil est le plus utilis\u00e9 est [sphinx](https://www.sphinx-doc.org/en/master/). Saurez-vous l'utiliser ?" ] }, { diff --git a/_unittests/ut_pycode/test_doc_helper.py b/_unittests/ut_pycode/test_doc_helper.py new file mode 100644 index 000000000..7765131a8 --- /dev/null +++ b/_unittests/ut_pycode/test_doc_helper.py @@ -0,0 +1,56 @@ +""" +@brief test tree node (time=7s) +""" +import unittest +import os +from pyquickhelper.pycode import ExtTestCase, ignore_warnings +from pyquickhelper.pycode.doc_helper import ( + find_link, validate_urls, validate_urls_in_folder) + + +class TestDocHelper(ExtTestCase): + + text = """ + + `zoo `_ + + `zoo `_ + + `zoo `_ + + .. image:: http://www.xavierdupre.fr/app/ensae_teaching_cs/helpsphinx/_static/project_ico.png + + .. download:: http://www.xavierdupre.fr/app/ensae_teaching_cs/helpsphinx/_static/project_ico.png + """ + + exp = [ + "https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/neural_network/_base.py", + "https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/neural_network/_base1.py", + "https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/neural_network/_base2.py", + "http://www.xavierdupre.fr/app/ensae_teaching_cs/helpsphinx/_static/project_ico.png", + "http://www.xavierdupre.fr/app/ensae_teaching_cs/helpsphinx/_static/project_ico.png", + ] + + def test_find_link(self): + res = find_link(TestDocHelper.text) + self.assertEqual(len(res), len(TestDocHelper.exp)) + for i in range(len(res)): + self.assertEqual(TestDocHelper.exp[i], res[i]) + + def test_validate_url(self): + val = validate_urls(TestDocHelper.exp) + self.assertEqual(len(val), 2) + + @ignore_warnings(ResourceWarning) + def test_validate_url_folder(self): + this = os.path.abspath(os.path.dirname(__file__)) + issues = [] + for issue in validate_urls_in_folder(this): + issues.append(issue) + self.assertEqual(len(issues), 2) + + +if __name__ == "__main__": + unittest.main() diff --git a/src/pyquickhelper/cli/encryption_cli.py b/src/pyquickhelper/cli/encryption_cli.py index 339b8576c..77486e851 100644 --- a/src/pyquickhelper/cli/encryption_cli.py +++ b/src/pyquickhelper/cli/encryption_cli.py @@ -57,7 +57,7 @@ def do_main(source, dest, password, encrypt, # pylint: disable=W0621 Encrypts or decrypts a folder, see @see cl EncryptedBackup. The function relies on module :epkg:`pycrypto`, :epkg:`cryptography`, algoritm `AES `_, - `Fernet `_. + `Fernet `_. @param source source of files to encrypt or decrypt @param dest destination @@ -114,7 +114,7 @@ def encrypt(fLOG=print, args=None): Encrypts using class @see cl EncryptedBackup. The function relies on module :epkg:`pycrypto`, :epkg:`cryptography`, algoritm `AES `_, - `Fernet `_. + `Fernet `_. @param fLOG logging function @param args to overwrite ``sys.args`` @@ -146,7 +146,7 @@ def decrypt(fLOG=print, args=None): Decrypts using class @see cl EncryptedBackup. The function relies on module :epkg:`pycrypto`, :epkg:`cryptography`, algoritm `AES `_, - `Fernet `_. + `Fernet `_. @param fLOG logging function @param args to overwrite ``sys.args`` diff --git a/src/pyquickhelper/filehelper/encryption.py b/src/pyquickhelper/filehelper/encryption.py index 2f812c748..e6aeb55e9 100644 --- a/src/pyquickhelper/filehelper/encryption.py +++ b/src/pyquickhelper/filehelper/encryption.py @@ -138,7 +138,7 @@ def encrypt_stream(key, filename, out_filename=None, chunksize=2 ** 18, algo="AE Encrypts a file using AES (CBC mode) with the given key. The function relies on module :epkg:`pycrypto`, :epkg:`cryptography`, algoritm `AES `_, - `Fernet `_. + `Fernet `_. @param key The encryption key - a string that must be either 16, 24 or 32 bytes long. Longer keys @@ -181,7 +181,7 @@ def decrypt_stream(key, filename, out_filename=None, chunksize=3 * 2 ** 13, algo Decrypts a file using AES (CBC mode) with the given key. The function relies on module :epkg:`pycrypto`, :epkg:`cryptography`, algoritm `AES `_, - `Fernet `_. + `Fernet `_. @param key The encryption key - a string that must be either 16, 24 or 32 bytes long. Longer keys diff --git a/src/pyquickhelper/filehelper/synchelper.py b/src/pyquickhelper/filehelper/synchelper.py index cb4c56def..b38773375 100644 --- a/src/pyquickhelper/filehelper/synchelper.py +++ b/src/pyquickhelper/filehelper/synchelper.py @@ -119,7 +119,7 @@ def listdir_aswalk(folder): def explore_folder_iterfile(folder, pattern=None, neg_pattern=None, - fullname=False, recursive=True): + fullname=False, recursive=True, verbose=False): """ Same as @see fn explore_folder but iterates on files included in a folder and its subfolders. @@ -130,6 +130,7 @@ def explore_folder_iterfile(folder, pattern=None, neg_pattern=None, :param neg_pattern: negative pattern to exclude files :param fullname: if True, include the subfolder while checking the regex :param recursive: look into subfolders + :param verbose: use :epkg:`tqdm` to display a progress bar :return: iterator on files """ if pattern is not None: @@ -143,27 +144,40 @@ def listdir_aswalk(folder): iter = os.walk if recursive else listdir_aswalk + def itera(folder): + for r, _, f in iter(folder): + for a in f: + yield r, f, a + + if verbose: + from tqdm import tqdm + loop = tqdm(itera(folder)) + else: + loop = itera(folder) + rep = {} - for r, _, f in iter(folder): - for a in f: - temp = os.path.join(r, a) - if pattern is not None: - if fullname: - if not pattern.search(temp): - continue - else: - if not pattern.search(a): - continue - if neg_pattern is not None: - if fullname: - if neg_pattern.search(temp): - continue - else: - if neg_pattern.search(a): - continue - yield temp - r = os.path.split(temp)[0] - rep[r] = None + for r, f, a in loop: + if verbose: + loop.set_description(r) + temp = os.path.join(r, a) + if pattern is not None: + if fullname: + if not pattern.search(temp): + continue + else: + if not pattern.search(a): + continue + if neg_pattern is not None: + if fullname: + if neg_pattern.search(temp): + continue + else: + if neg_pattern.search(a): + print("---", temp) + continue + yield temp + r = os.path.split(temp)[0] + rep[r] = None def explore_folder_iterfile_repo(folder, log=fLOG): diff --git a/src/pyquickhelper/helpgen/default_conf.py b/src/pyquickhelper/helpgen/default_conf.py index 604de57d1..49c0041d9 100644 --- a/src/pyquickhelper/helpgen/default_conf.py +++ b/src/pyquickhelper/helpgen/default_conf.py @@ -73,7 +73,7 @@ def get_epkg_dictionary(): 'azure pipelines': 'https://azure.microsoft.com/en-us/services/devops/pipelines/', 'Azure Pipelines': 'https://azure.microsoft.com/en-us/services/devops/pipelines/', 'bokeh': 'https://bokeh.pydata.org/en/latest/', - 'builderapi': 'http://www.sphinx-doc.org/en/stable/extdev/builderapi.html', + 'builderapi': 'https://www.sphinx-doc.org/en/master/extdev/builderapi.html', 'bz2': 'https://en.wikipedia.org/wiki/Bzip2', 'cairosvg': 'https://github.com/Kozea/CairoSVG', 'chrome': 'https://www.google.com/chrome/', @@ -108,7 +108,7 @@ def get_epkg_dictionary(): 'Graphviz': 'http://www.graphviz.org/', 'html': 'https://en.wikipedia.org/wiki/HTML', 'HTML': 'https://en.wikipedia.org/wiki/HTML', - 'imgmath': 'http://www.sphinx-doc.org/en/master/usage/extensions/math.html#module-sphinx.ext.imgmath', + 'imgmath': 'https://www.sphinx-doc.org/en/master/usage/extensions/math.html#module-sphinx.ext.imgmath', 'img2pdf': 'https://gitlab.mister-muffin.de/josch/img2pdf', 'Inkscape': 'https://inkscape.org/', 'InkScape': 'https://inkscape.org/', @@ -202,18 +202,18 @@ def get_epkg_dictionary(): ('https://docs.scipy.org/doc/scipy/reference/generated/scipy.{0}.html', 1), ('https://docs.scipy.org/doc/scipy/reference/generated/scipy.{0}.{1}.html', 2)), 'SFTP': 'https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol', - 'sphinx': 'http://www.sphinx-doc.org/en/stable/', - 'Sphinx': 'http://www.sphinx-doc.org/en/stable/', - 'sphinx.ext.autodoc': 'http://www.sphinx-doc.org/en/stable/ext/autodoc.html#module-sphinx.ext.autodoc', + 'sphinx': 'https://www.sphinx-doc.org/en/master/', + 'Sphinx': 'https://www.sphinx-doc.org/en/master/', + 'sphinx.ext.autodoc': 'https://www.sphinx-doc.org/en/master/ext/autodoc.html#module-sphinx.ext.autodoc', 'sphinx.ext.intersphinx': 'https://www.sphinx-doc.org/en/master/usage/extensions/intersphinx.html', 'sphinx-gallery': 'https://sphinx-gallery.readthedocs.io/en/latest/', - 'Sphinx application': 'http://www.sphinx-doc.org/en/stable/_modules/sphinx/application.html', + 'Sphinx application': 'https://www.sphinx-doc.org/en/master/_modules/sphinx/application.html', 'svg': 'https://fr.wikipedia.org/wiki/Scalable_Vector_Graphics', 'SVG': 'https://fr.wikipedia.org/wiki/Scalable_Vector_Graphics', 'SVN': 'https://subversion.apache.org/', 'svn': 'https://subversion.apache.org/', 'tar.gz': 'https://en.wikipedia.org/wiki/Tar_(computing)', - 'toctree': 'http://www.sphinx-doc.org/en/stable/markup/toctree.html', + 'toctree': 'https://www.sphinx-doc.org/en/master/markup/toctree.html', 'TexnicCenter': 'http://www.texniccenter.org/', 'tinycss2': 'http://pythonhosted.org/tinycss2/', 'tkinter': 'https://docs.python.org/3/library/tkinter.html', @@ -270,8 +270,8 @@ def set_sphinx_variables(fileconf, module_name, author, year, theme, theme_path, sphinx-bootstrap-theme/README.html>`_ @param description_latex description latex @param use_mathjax set up the documentation to use mathjax, - see `sphinx.ext.mathjax `_, + see `sphinx.ext.mathjax + `_, default option is True @param use_lunrsearch suggest autocompletion in sphinx, see `sphinxcontrib-lunrsearch `_, + @param extlinks parameter `extlinks `_, example: ``{'issue': ('https://github.com/sdpython/pyquickhelper/issues/%s', 'issue ')}`` @param github_user git(hub) user @param github_repo git(hub) project @@ -614,7 +614,7 @@ def set_sphinx_variables(fileconf, module_name, author, year, theme, theme_path, # texinfo_show_urls = 'footnote' # it modifies the set of things to display inside the sidebar - # see http://www.sphinx-doc.org/en/stable/config.html#confval-html_sidebars + # see https://www.sphinx-doc.org/en/master/config.html#confval-html_sidebars html_sidebars = { '[!blog]**': ['searchbox.html', 'moduletoc.html', 'relations.html', 'sourcelink.html', ], 'blog/**': ['searchbox.html', 'blogtoc.html', 'localtoc.html', 'sourcelink.html', ], @@ -903,7 +903,8 @@ def get_first_line(filename): def _skip(app, what, name, obj, skip, options): """ To skip some functions, - see `Skipping members `_. + see `Skipping members + `_. """ if name.startswith("_") and name not in \ ["__qualname__", diff --git a/src/pyquickhelper/helpgen/rst_converters.py b/src/pyquickhelper/helpgen/rst_converters.py index fe3c00e6d..28441d5d9 100644 --- a/src/pyquickhelper/helpgen/rst_converters.py +++ b/src/pyquickhelper/helpgen/rst_converters.py @@ -117,7 +117,7 @@ def rst2html(s, fLOG=noLOG, writer="html", keep_warnings=False, @param destination set a destination (requires for some extension) @param destination_path set a destination path (requires for some extension) @param options :epkg:`Sphinx` options see - `Render math as images `_, + `Render math as images `_, a subset of options is used, see @see fn default_sphinx_options. By default, the theme (option *html_theme*) will ``'basic'``. @return HTML format @@ -125,7 +125,8 @@ def rst2html(s, fLOG=noLOG, writer="html", keep_warnings=False, *directives* is None or a list of 2 or 5-uple: * a directive name (mandatory) - * a directive class: see `Sphinx Directive `_, + * a directive class: see `Sphinx Directive + `_, see also @see cl RunPythonDirective as an example (mandatory) * a docutils node: see @see cl runpython_node as an example * two functions: see @see fn visit_runpython_node, @see fn depart_runpython_node as an example @@ -477,7 +478,7 @@ def docstring2html(function_or_string, format="html", fLOG=noLOG, writer="html", @param filter_nodes transform the doctree before writing the results (layout must be 'sphinx') @param options Sphinx options see `Render math as images - `_, + `_, a subset of options is used, see @see fn default_sphinx_options. By default, the theme (option *html_theme*) will ``'basic'``. @return (str) :epkg:`HTML` format or (IPython.core.display.HTML) diff --git a/src/pyquickhelper/helpgen/sphinx_main.py b/src/pyquickhelper/helpgen/sphinx_main.py index 30526d085..e043b76d4 100644 --- a/src/pyquickhelper/helpgen/sphinx_main.py +++ b/src/pyquickhelper/helpgen/sphinx_main.py @@ -233,7 +233,7 @@ def generate_help_sphinx(project_var_name, clean=False, root=".", script which outputs RST documntation adds it to the current documentation. The function automatically adds custom role and custom directive ``sharenet``. The function directly calls - `sphinx `_, + `sphinx `_, `nbconvert `_. When there are too many notebooks, the notebook index is difficult to read. It does not require to get script location. diff --git a/src/pyquickhelper/helpgen/sphinxm_convert_doc_sphinx_helper.py b/src/pyquickhelper/helpgen/sphinxm_convert_doc_sphinx_helper.py index 9ac33cea1..45c62ef95 100644 --- a/src/pyquickhelper/helpgen/sphinxm_convert_doc_sphinx_helper.py +++ b/src/pyquickhelper/helpgen/sphinxm_convert_doc_sphinx_helper.py @@ -865,7 +865,7 @@ def _get_filename(self, targetname, encoding='utf-8', overwrite_if_changed=True) class _CustomBuildEnvironment(BuildEnvironment): """ Overrides some functionalities of - `BuildEnvironment `_. + `BuildEnvironment `_. """ def __init__(self, app): diff --git a/src/pyquickhelper/helpgen/sphinxm_mock_app.py b/src/pyquickhelper/helpgen/sphinxm_mock_app.py index d26c8ebda..922ccfb41 100644 --- a/src/pyquickhelper/helpgen/sphinxm_mock_app.py +++ b/src/pyquickhelper/helpgen/sphinxm_mock_app.py @@ -275,7 +275,8 @@ def create(writer="html", directives=None, confoverrides=None, *directives* is None or a list of 2 or 5-uple: * a directive name (mandatory) - * a directive class: see `Sphinx Directive `_, + * a directive class: see `Sphinx Directive + `_, see also @see cl RunPythonDirective as an example (mandatory) * a docutils node: see @see cl runpython_node as an example * two functions: see @see fn visit_runpython_node, diff --git a/src/pyquickhelper/ipythonhelper/notebook_helper.py b/src/pyquickhelper/ipythonhelper/notebook_helper.py index 17ae60368..94ffe69ff 100644 --- a/src/pyquickhelper/ipythonhelper/notebook_helper.py +++ b/src/pyquickhelper/ipythonhelper/notebook_helper.py @@ -465,7 +465,7 @@ def remove_execution_number(infile, outfile=None, encoding="utf-8", indent=2, ru Remove execution number from the notebook to avoid commiting changes only about those numbers - `notebook 5.1.0 `_ + `notebook 5.1.0 `_ introduced changes which are incompatible with leaving the cell executing number empty. """ diff --git a/src/pyquickhelper/pycode/doc_helper.py b/src/pyquickhelper/pycode/doc_helper.py new file mode 100644 index 000000000..ec54e5c1a --- /dev/null +++ b/src/pyquickhelper/pycode/doc_helper.py @@ -0,0 +1,69 @@ +""" +@file +@brief Helpers to improve documentation. +""" +import re +from urllib.request import urlopen +from ..filehelper.synchelper import explore_folder_iterfile + + +def find_link(text): + """ + Finds all links following RST format in a documentation. + + :param text: text + :return: all urls + """ + url = "https?://[-a-zA-Z0-9@:%._\+~#=]+?[-a-zA-Z0-9@:%._\+~#=/&?\\n ]*?" + reg = [re.compile("[<](%s)[>]" % url), + re.compile("[.]{2} image:: (%s)\\n" % url), + re.compile("[.]{2} download:: (%s)\\n" % url)] + res = [] + for r in reg: + a = r.findall(text) + if len(a) > 0: + res.extend([_.replace("\n", "").replace(" ", "") for _ in a]) + return res + + +def validate_urls(urls): + """ + Checks that all urls are valid. + """ + issue = [] + for u in urls: + try: + with urlopen(u, timeout=10) as f: + content = f.read(10) + if len(content) != 10: + issue.append((u, "Cannot download")) + except Exception as e: + issue.append((u, e)) + return issue + + +def validate_urls_in_folder(folder, ext="py,rst,ipynb", + neg_pattern=".*__pycache__.*", + recursive=True, verbose=False): + """ + Looks for all files in a folder and return all invalid urls. + + :param folder: folder to look into + :param ext: files extension to look into + :param neg_pattern: exclude files following that pattern + :param recursive: look into sub folders + :param verbose: use :epkg:`tqdm` to display a progress bar + :return: enumerator on issues + """ + if isinstance(ext, str): + ext = ext.split(",") + pattern = ".*[.](%s)$" % "|".join(["(%s)" % e for e in ext]) + for name in explore_folder_iterfile( + folder, pattern=pattern, neg_pattern=None, + fullname=True, recursive=recursive, verbose=verbose): + with open(name, "r", encoding="utf-8") as f: + content = f.read() + urls = find_link(content) + issues = validate_urls(urls) + for issue in issues: + yield (name, ) + issue diff --git a/src/pyquickhelper/pycode/utils_tests.py b/src/pyquickhelper/pycode/utils_tests.py index f28989c33..ddd46d93d 100644 --- a/src/pyquickhelper/pycode/utils_tests.py +++ b/src/pyquickhelper/pycode/utils_tests.py @@ -133,10 +133,8 @@ def main_wrapper_tests(logfile, skip_list=None, processes=False, add_coverage=Fa replace full paths by relative path. Parameters *coverage_options*, *coverage_exclude_lines*, *additional_ut_path*: - see class `Coverage `_ - and `Configuration files `_ + see class `Coverage `_ + and `Configuration files `_ to specify those options. If both values are left to None, this function will compute the code coverage for all files in this module. The function now exports the coverage options which were used. diff --git a/src/pyquickhelper/pycode/utils_tests_helper.py b/src/pyquickhelper/pycode/utils_tests_helper.py index d778d7a44..0d2ffd6f5 100644 --- a/src/pyquickhelper/pycode/utils_tests_helper.py +++ b/src/pyquickhelper/pycode/utils_tests_helper.py @@ -155,9 +155,8 @@ def check_pep8(folder, ignore=('E265', 'W504'), skip=None, :epkg:`PEP8` is not verified, see also `Error Codes `_ @param pylint_ignore ignore :epkg:`pylint` issues, see - `pylint error codes `_ - @param complexity see `check_file `_ + :epkg:`pylint error codes` + @param complexity see `check_file `_ @param stop_after stop after *stop_after* issues @param skip skip a warning if a substring in this list is found @param neg_pattern skip files verifying this regular expressions diff --git a/src/pyquickhelper/sphinxext/blog_post.py b/src/pyquickhelper/sphinxext/blog_post.py index ca2c7d2bb..3e459fa0a 100644 --- a/src/pyquickhelper/sphinxext/blog_post.py +++ b/src/pyquickhelper/sphinxext/blog_post.py @@ -3,9 +3,7 @@ @file @brief Helpers to process blog post included in the documentation. """ - import os -import sys from io import StringIO from contextlib import redirect_stdout, redirect_stderr from docutils import io as docio @@ -139,7 +137,7 @@ def __init__(self, filename, encoding='utf-8-sig', raise_exception=False, extens "Unable to parse a blogpost:\n[sphinxerror]-F\n{0}" "\nFILE\n{1}\nCONTENT\n{2}\n--OUT--\n{3}".format( all_err, self._filename, content, keepout.getvalue())) - elif len(lines) > 0: + if len(lines) > 0: print(all_err) if len(std) > 3: print(std) diff --git a/src/pyquickhelper/sphinxext/sphinx_blog_extension.py b/src/pyquickhelper/sphinxext/sphinx_blog_extension.py index 2e961ee1b..2cdb16a80 100644 --- a/src/pyquickhelper/sphinxext/sphinx_blog_extension.py +++ b/src/pyquickhelper/sphinxext/sphinx_blog_extension.py @@ -2,8 +2,10 @@ """ @file @brief Defines blogpost directives. -See `Tutorial: Writing a simple extension `_, -`Creating reStructuredText Directives `_ +See `Tutorial: Writing a simple extension +`_, +`Creating reStructuredText Directives +`_ """ import os import sphinx diff --git a/src/pyquickhelper/sphinxext/sphinx_epkg_extension.py b/src/pyquickhelper/sphinxext/sphinx_epkg_extension.py index 3ffb2d545..c2d22347d 100644 --- a/src/pyquickhelper/sphinxext/sphinx_epkg_extension.py +++ b/src/pyquickhelper/sphinxext/sphinx_epkg_extension.py @@ -37,7 +37,7 @@ def epkg_role(role, rawtext, text, lineno, inliner, options=None, content=None): :: - `to_html `_ + `to_html `_ By: diff --git a/src/pyquickhelper/sphinxext/sphinx_md_builder.py b/src/pyquickhelper/sphinxext/sphinx_md_builder.py index 2227c8d0b..094387c95 100644 --- a/src/pyquickhelper/sphinxext/sphinx_md_builder.py +++ b/src/pyquickhelper/sphinxext/sphinx_md_builder.py @@ -3,7 +3,7 @@ @file @brief Defines a sphinx extension to output the documentation in :epkg:`Markdown` or *MD*. It is inspired from `restbuilder -`_. +`_. I replicate its license here: :: diff --git a/src/pyquickhelper/sphinxext/sphinx_rst_builder.py b/src/pyquickhelper/sphinxext/sphinx_rst_builder.py index ada1681fa..aa4a85fd1 100644 --- a/src/pyquickhelper/sphinxext/sphinx_rst_builder.py +++ b/src/pyquickhelper/sphinxext/sphinx_rst_builder.py @@ -3,7 +3,7 @@ @file @brief Defines a sphinx extension to output the documentation in :epkg:`RST`. It is inspired from `restbuilder -`_. +`_. I replicate its license here: :: diff --git a/src/pyquickhelper/sphinxext/sphinx_runpython_extension.py b/src/pyquickhelper/sphinxext/sphinx_runpython_extension.py index 88e09be72..5b8bf4263 100644 --- a/src/pyquickhelper/sphinxext/sphinx_runpython_extension.py +++ b/src/pyquickhelper/sphinxext/sphinx_runpython_extension.py @@ -2,7 +2,8 @@ """ @file @brief Defines runpython directives. -See `Tutorial: Writing a simple extension `_ +See `Tutorial: Writing a simple extension +`_ """ import sys import os @@ -328,7 +329,8 @@ class RunPythonDirective(Directive): * ``:showout`` if *:rst:* is set up, this flag adds the raw rst output to check what is happening * ``:sin:`` which text to display before the code (by default *In*) * ``:sout:`` which text to display before the output (by default *Out*) - * ``:sphinx:`` by default, function `nested_parse_with_titles `_ is + * ``:sphinx:`` by default, function `nested_parse_with_titles + `_ is used to parse the output of the script, if this option is set to false, `public_doctree `_. * ``:store:`` stores the local context in :epkg:`sphinx` application to restore it later @@ -365,7 +367,8 @@ class RunPythonDirective(Directive): If the directive produces RST text to be included later in the documentation, it is able to interpret `docutils directives `_ - and `Sphinx directives `_ + and `Sphinx directives + `_ with function `nested_parse_with_titles `_. However, if this text contains titles, it is better to use option ``:sphinx: false``. diff --git a/src/pyquickhelper/sphinxext/sphinx_toctree_extension.py b/src/pyquickhelper/sphinxext/sphinx_toctree_extension.py index d6ac1330f..6c47beb54 100644 --- a/src/pyquickhelper/sphinxext/sphinx_toctree_extension.py +++ b/src/pyquickhelper/sphinxext/sphinx_toctree_extension.py @@ -1,7 +1,7 @@ # -*- coding: utf-8 -*- """ @file -@brief Overwrites `toctree `_ +@brief Overwrites `toctree `_ directive to get catch exceptions when a document is processed inline. """ from docutils.parsers.rst import directives @@ -22,7 +22,7 @@ class CustomTocTree(TocTree): """ Overwrites `toctree - `_. + `_. The code is located at `sphinx/directives/other.py `_. diff --git a/src/pyquickhelper/sphinxext/sphinx_todoext_extension.py b/src/pyquickhelper/sphinxext/sphinx_todoext_extension.py index 0e19447cc..acd5550b1 100644 --- a/src/pyquickhelper/sphinxext/sphinx_todoext_extension.py +++ b/src/pyquickhelper/sphinxext/sphinx_todoext_extension.py @@ -41,7 +41,7 @@ class TodoExt(BaseAdmonition): * *title:* a title for the todo (mandatory) * *tag:* a tag to have several categories of todo (mandatory) - * *issue:* the issue requires `extlinks `_ + * *issue:* the issue requires `extlinks `_ to be defined and must contain key ``'issue'`` (optional) * *cost:* a cost if the todo were to be fixed (optional) * *priority:* to prioritize items (optional) @@ -500,7 +500,7 @@ def setup(app): app.add_config_value('todoext_link_only', False, 'html') # The following variable is shared with extension - # `todo `_. + # `todo `_. try_add_config_value(app, 'extlinks', {}, 'env') app.add_node(todoextlist, diff --git a/src/pyquickhelper/sphinxext/sphinxext_helper.py b/src/pyquickhelper/sphinxext/sphinxext_helper.py index 0fbdabe04..0f666d4e4 100644 --- a/src/pyquickhelper/sphinxext/sphinxext_helper.py +++ b/src/pyquickhelper/sphinxext/sphinxext_helper.py @@ -17,7 +17,7 @@ def try_add_config_value(app, name, default, rebuild, type_s=()): @return True if added, False if already present. Rebuilds can be (source: `Sphinx.add_config_value - `_): + `_): * 'env' if a change in the setting only takes effect when a document is parsed - this means that the whole environment must be rebuilt. diff --git a/src/quicksetup/pyquicksetup/__init__.py b/src/quicksetup/pyquicksetup/__init__.py index 7ccbda727..60ea00441 100644 --- a/src/quicksetup/pyquicksetup/__init__.py +++ b/src/quicksetup/pyquicksetup/__init__.py @@ -24,6 +24,7 @@ SetupCommandUnitTestGUI, SetupCommandUnitTestLONG, SetupCommandUnitTestSKIP, + SetupCommandValidateUrls, SetupCommandVersion) from .helper import _clean_readme @@ -44,6 +45,7 @@ def default_cmdclass(): 'unittests_GUI': SetupCommandUnitTestGUI, 'unittests_LONG': SetupCommandUnitTestLONG, 'unittests_SKIP': SetupCommandUnitTestSKIP, + 'validate_urls': SetupCommandValidateUrls, 'write_version': SetupCommandVersion} diff --git a/src/quicksetup/pyquicksetup/pyquick.py b/src/quicksetup/pyquicksetup/pyquick.py index cc7fa973a..599a76601 100644 --- a/src/quicksetup/pyquicksetup/pyquick.py +++ b/src/quicksetup/pyquicksetup/pyquick.py @@ -307,6 +307,38 @@ def run(self): process_standard_options_for_setup(**parameters) +class SetupCommandValidateUrls(_SetupCommand): + description = "Validate Urls in the documentation." + + user_options = [ + ('folder=', 'f', 'folder to look into'), + ('verbose=', 'v', 'verbose'), + ] + + def initialize_options(self): + self.folder = "." + self.verbose = False + + def finalize_options(self): + pass + + def run(self): + from pyquickhelper.pycode.doc_helper import validate_urls_in_folder + parameters = self.get_parameters() + parameters['argv'] = ['unittests'] + if self.folder is None: + folder = '.' + else: + folder = self.folder + if self.verbose is None: + verbose = False + else: + verbose = True + for issue in validate_urls_in_folder(folder=folder, verbose=verbose): + print("ERROR url=%r in %r error=%r" % + (issue[1], issue[0], issue[2])) + + class SetupCommandVersion(_SetupCommand): description = ( "Retrieves the commit number from git and writes it "