Adds a command in the setup to check url in the code or documentation

_doc/sphinxdoc/source/blog/2015/2015-10-10_tips_sphinx.rst

@@ -30,12 +30,12 @@
 
     **latex formulas**
 
-    The page `Math support in Sphinx <http://sphinx-doc.org/ext/math.html?highlight=math#module-sphinx.ext.mathbase>`_
+    The page `Math support in Sphinx <https://www.sphinx-doc.org/en/master/usage/extensions/math.html>`_
     explains how to set up math environment (latex or mathjax).
     But if you add matplotlib to convert equations into images
     (`matplotlib.sphinxext.mathmpl <http://matplotlib.org/sampledoc/extensions.html#using-math>`_),
     the sphinx extension
-    `sphinx.ext.pngmath is disabled <http://sphinx-doc.org/ext/math.html#module-sphinx.ext.pngmath>`_.
+    `sphinx.ext.pngmath is disabled <https://www.sphinx-doc.org/en/master/#module-sphinx.ext.pngmath>`_.
     Matplotlib extension has some limitations. ``\text`` does not work.
 
     **formulas in docstring**

_doc/sphinxdoc/source/blog/2015/2015-12-12_sphinx_extensions.rst

@@ -6,7 +6,7 @@
     :categories: sphinx
 
     The following repository
-    `birkenfeld/sphinx-contrib/ <https://bitbucket.org/birkenfeld/sphinx-contrib/src/284d5b04263c07857bbc3cf743136f9cfba0f170?at=default>`_
+    `birkenfeld/sphinx-contrib/ <https://github.com/sphinx-contrib/legacy>`_
     contains many useful extensions
     to improve the rendering of
     `Sphinx <http://sphinx-doc.org/>`_ documentation:

_doc/sphinxdoc/source/blog/2017/2017-05-17_sphinx.rst

@@ -5,7 +5,7 @@
     :date: 2017-05-17
     :categories: documentation
 
-    `Sphinx <http://www.sphinx-doc.org/en/stable/>`_
+    `Sphinx <https://www.sphinx-doc.org/en/master/>`_
     has released a new version 1.6.1 but it breaks a couple of
     packages.
     `sphinx_gallery <https://sphinx-gallery.readthedocs.io/en/latest/>`_

_doc/sphinxdoc/source/blog/2017/2017-05-21_sphinx.rst

@@ -5,7 +5,7 @@
     :date: 2017-05-21
     :categories: documentation
 
-    `Sphinx <http://www.sphinx-doc.org/en/stable/>`_
+    `Sphinx <https://www.sphinx-doc.org/en/master/>`_
     has released a new version 1.6.1. Other packages
     updated their code but some issues remain.
     I had to modify the code of the file

_doc/sphinxdoc/source/blog/2017/2017-08-25_sphinxgallery.rst

@@ -12,7 +12,7 @@
     There is a discussion about it on
     `scikit-learn/9189 <https://github.com/scikit-learn/scikit-learn/issues/9189>`_
     and the package is now included in the
-    `sources <https://github.com/scikit-learn/scikit-learn/tree/master/doc/sphinxext/sphinx_gallery>`_.
+    `sources <https://scikit-learn.org/stable/auto_examples/index.html>`_.
     It is not really bothering for a small package like this.
     But still, it is extra work.
     I realize open source holds on some kind of magic

_doc/sphinxdoc/source/contribute.rst

@@ -282,7 +282,8 @@ It relies on epkg:`pyquickhelper`.
 Generation
 ++++++++++
 
-The documentation can be written using `RST <http://sphinx-doc.org/rest.html>`_ format
+The documentation can be written using `RST
+<https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html>`_ format
 or `javadoc <http://en.wikipedia.org/wiki/Javadoc>`_ format. The documentation
 can generated by:
 

_doc/sphinxdoc/source/glossary.rst

@@ -11,6 +11,6 @@ Glossary
 
     RST or ReSTructured Text
         This format is used to write the documentation of most of Python module.
-        The module `Sphinx <http://www.sphinx-doc.org/en/>`_ compiles it into
+        The module `Sphinx <https://www.sphinx-doc.org/en/master/>`_ compiles it into
         a HTML pages. See `RST syntax <http://openalea.gforge.inria.fr/doc/openalea/doc/_build/html/source/sphinx/rest_syntax.html>`_
         or `RST cheat sheet <https://github.com/ralsina/rst-cheatsheet/blob/master/rst-cheatsheet.rst>`_.

_doc/sphinxdoc/source/tutorial/sphinx.rst

@@ -20,7 +20,7 @@ Simple extensions
 -----------------
 
 :epkg:`Sphinx` implements many
-`markups <http://www.sphinx-doc.org/en/stable/markup/index.html#sphinxmarkup>`_.
+`markups <https://www.sphinx-doc.org/en/master/markup/index.html#sphinxmarkup>`_.
 This module adds a couple of them. Many cheat sheets
 (see `cheat sheet 1 <https://thomas-cokelaer.info/tutorials/sphinx/rest_syntax.html>`_,
 `cheat sheet 2 <http://docs.sphinxdocs.com/en/latest/cheatsheet.html>`_,
@@ -584,7 +584,7 @@ does and a little bit more:
 
 This design is described by an empty module:
 
-* `documentation <http://www.xavierdupre.fr/app/python3_module_template/helpsphinx2/index.html>`_
+* `documentation <http://www.xavierdupre.fr/app/python3_module_template/helpsphinx/index.html>`_
 * `github/python3_module_template <https://github.com/sdpython/python3_module_template/>`_
 
 Blog Post

_unittests/ut_cli/data/glossary.rst

@@ -11,6 +11,6 @@ Glossary
 
     RST or ReSTructured Text
         This format is used to write the documentation of most of Python module.
-        The module `Sphinx <http://www.sphinx-doc.org/en/>`_ compiles it into
+        The module `Sphinx <https://www.sphinx-doc.org/en/master/>`_ compiles it into
         a HTML pages. See `RST syntax <http://openalea.gforge.inria.fr/doc/openalea/doc/_build/html/source/sphinx/rest_syntax.html>`_
         or `RST cheat sheet <https://github.com/ralsina/rst-cheatsheet/blob/master/rst-cheatsheet.rst>`_.

_unittests/ut_cli/data/td1a_unit_test_ci.ipynb

@@ -438,7 +438,7 @@
       "source": [
         "## Ecrire la documentation\n",
         "\n",
-        "L'outil est le plus utilis\u00e9 est [sphinx](http://www.sphinx-doc.org/en/stable/). Saurez-vous l'utiliser ?"
+        "L'outil est le plus utilis\u00e9 est [sphinx](https://www.sphinx-doc.org/en/master/). Saurez-vous l'utiliser ?"
       ]
     },
     {

_unittests/ut_helpgen/data_project/pp/_doc/sphinxdoc/source/index.rst

@@ -45,7 +45,7 @@ python3_module_template
         :alt: size
 
 **Links:** `github <https://github.com/sdpython/python3_module_template/>`_,
-`documentation <http://www.xavierdupre.fr/app/python3_module_template/helpsphinx2/index.html>`_,
+`documentation <http://www.xavierdupre.fr/app/python3_module_template/helpsphinx/index.html>`_,
 `travis <https://travis-ci.org/sdpython/python3_module_template>`_,
 :ref:`l-README`,
 :ref:`blog <ap-main-0>`,

_unittests/ut_helpgen/notebooks_svg/td1a_unit_test_ci.ipynb

@@ -437,7 +437,7 @@
       "source": [
         "## Ecrire la documentation\n",
         "\n",
-        "L'outil est le plus utilis\u00e9 est [sphinx](http://www.sphinx-doc.org/en/stable/). Saurez-vous l'utiliser ?"
+        "L'outil est le plus utilis\u00e9 est [sphinx](https://www.sphinx-doc.org/en/master/). Saurez-vous l'utiliser ?"
       ]
     },
     {

_unittests/ut_pycode/test_doc_helper.py

@@ -0,0 +1,56 @@
+"""
+@brief      test tree node (time=7s)
+"""
+import unittest
+import os
+from pyquickhelper.pycode import ExtTestCase, ignore_warnings
+from pyquickhelper.pycode.doc_helper import (
+    find_link, validate_urls, validate_urls_in_folder)
+
+
+class TestDocHelper(ExtTestCase):
+
+    text = """
+
+    `zoo <https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/neural_network/_base.py>`_
+
+    `zoo <https://github.com/scikit-learn/scikit-learn/
+    blob/main/sklearn/neural_network/_base1.py>`_
+
+    `zoo <https://github.com/scikit-learn/scikit-learn/
+    blob/main/sklearn/neural_network/ _base2.py>`_
+
+    .. image:: http://www.xavierdupre.fr/app/ensae_teaching_cs/helpsphinx/_static/project_ico.png
+
+    .. download:: http://www.xavierdupre.fr/app/ensae_teaching_cs/helpsphinx/_static/project_ico.png
+    """
+
+    exp = [
+        "https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/neural_network/_base.py",
+        "https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/neural_network/_base1.py",
+        "https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/neural_network/_base2.py",
+        "http://www.xavierdupre.fr/app/ensae_teaching_cs/helpsphinx/_static/project_ico.png",
+        "http://www.xavierdupre.fr/app/ensae_teaching_cs/helpsphinx/_static/project_ico.png",
+    ]
+
+    def test_find_link(self):
+        res = find_link(TestDocHelper.text)
+        self.assertEqual(len(res), len(TestDocHelper.exp))
+        for i in range(len(res)):
+            self.assertEqual(TestDocHelper.exp[i], res[i])
+
+    def test_validate_url(self):
+        val = validate_urls(TestDocHelper.exp)
+        self.assertEqual(len(val), 2)
+
+    @ignore_warnings(ResourceWarning)
+    def test_validate_url_folder(self):
+        this = os.path.abspath(os.path.dirname(__file__))
+        issues = []
+        for issue in validate_urls_in_folder(this):
+            issues.append(issue)
+        self.assertEqual(len(issues), 2)
+
+
+if __name__ == "__main__":
+    unittest.main()

src/pyquickhelper/cli/encryption_cli.py

@@ -57,7 +57,7 @@ def do_main(source, dest, password, encrypt,  # pylint: disable=W0621
     Encrypts or decrypts a folder, see @see cl EncryptedBackup.
     The function relies on module :epkg:`pycrypto`, :epkg:`cryptography`,
     algoritm `AES <https://fr.wikipedia.org/wiki/Advanced_Encryption_Standard>`_,
-    `Fernet <http://cryptography.readthedocs.org/en/latest/fernet/>`_.
+    `Fernet <https://cryptography.io/en/latest/fernet/>`_.
 
     @param      source      source of files to encrypt or decrypt
     @param      dest        destination
@@ -114,7 +114,7 @@ def encrypt(fLOG=print, args=None):
     Encrypts using class @see cl EncryptedBackup.
     The function relies on module :epkg:`pycrypto`, :epkg:`cryptography`,
     algoritm `AES <https://fr.wikipedia.org/wiki/Advanced_Encryption_Standard>`_,
-    `Fernet <http://cryptography.readthedocs.org/en/latest/fernet/>`_.
+    `Fernet <https://cryptography.io/en/latest/fernet/>`_.
 
     @param      fLOG        logging function
     @param      args        to overwrite ``sys.args``
@@ -146,7 +146,7 @@ def decrypt(fLOG=print, args=None):
     Decrypts using class @see cl EncryptedBackup.
     The function relies on module :epkg:`pycrypto`, :epkg:`cryptography`,
     algoritm `AES <https://fr.wikipedia.org/wiki/Advanced_Encryption_Standard>`_,
-    `Fernet <http://cryptography.readthedocs.org/en/latest/fernet/>`_.
+    `Fernet <https://cryptography.io/en/latest/fernet/>`_.
 
     @param      fLOG        logging function
     @param      args        to overwrite ``sys.args``

src/pyquickhelper/filehelper/encryption.py

@@ -138,7 +138,7 @@ def encrypt_stream(key, filename, out_filename=None, chunksize=2 ** 18, algo="AE
     Encrypts a file using AES (CBC mode) with the given key.
     The function relies on module :epkg:`pycrypto`, :epkg:`cryptography`,
     algoritm `AES <https://fr.wikipedia.org/wiki/Advanced_Encryption_Standard>`_,
-    `Fernet <http://cryptography.readthedocs.org/en/latest/fernet/>`_.
+    `Fernet <https://cryptography.io/en/latest/fernet/>`_.
 
     @param      key             The encryption key - a string that must be
                                 either 16, 24 or 32 bytes long. Longer keys
@@ -181,7 +181,7 @@ def decrypt_stream(key, filename, out_filename=None, chunksize=3 * 2 ** 13, algo
     Decrypts a file using AES (CBC mode) with the given key.
     The function relies on module :epkg:`pycrypto`, :epkg:`cryptography`,
     algoritm `AES <https://fr.wikipedia.org/wiki/Advanced_Encryption_Standard>`_,
-    `Fernet <http://cryptography.readthedocs.org/en/latest/fernet/>`_.
+    `Fernet <https://cryptography.io/en/latest/fernet/>`_.
 
     @param      key             The encryption key - a string that must be
                                 either 16, 24 or 32 bytes long. Longer keys

src/pyquickhelper/filehelper/synchelper.py

@@ -119,7 +119,7 @@ def listdir_aswalk(folder):
 
 
 def explore_folder_iterfile(folder, pattern=None, neg_pattern=None,
-                            fullname=False, recursive=True):
+                            fullname=False, recursive=True, verbose=False):
     """
     Same as @see fn explore_folder but iterates on files
     included in a folder and its subfolders.
@@ -130,6 +130,7 @@ def explore_folder_iterfile(folder, pattern=None, neg_pattern=None,
     :param neg_pattern: negative pattern to exclude files
     :param fullname: if True, include the subfolder while checking the regex
     :param recursive: look into subfolders
+    :param verbose: use :epkg:`tqdm` to display a progress bar
     :return: iterator on files
     """
     if pattern is not None:
@@ -143,27 +144,40 @@ def listdir_aswalk(folder):
 
     iter = os.walk if recursive else listdir_aswalk
 
+    def itera(folder):
+        for r, _, f in iter(folder):
+            for a in f:
+                yield r, f, a
+
+    if verbose:
+        from tqdm import tqdm
+        loop = tqdm(itera(folder))
+    else:
+        loop = itera(folder)
+
     rep = {}
-    for r, _, f in iter(folder):
-        for a in f:
-            temp = os.path.join(r, a)
-            if pattern is not None:
-                if fullname:
-                    if not pattern.search(temp):
-                        continue
-                else:
-                    if not pattern.search(a):
-                        continue
-            if neg_pattern is not None:
-                if fullname:
-                    if neg_pattern.search(temp):
-                        continue
-                else:
-                    if neg_pattern.search(a):
-                        continue
-            yield temp
-            r = os.path.split(temp)[0]
-            rep[r] = None
+    for r, f, a in loop:
+        if verbose:
+            loop.set_description(r)
+        temp = os.path.join(r, a)
+        if pattern is not None:
+            if fullname:
+                if not pattern.search(temp):
+                    continue
+            else:
+                if not pattern.search(a):
+                    continue
+        if neg_pattern is not None:
+            if fullname:
+                if neg_pattern.search(temp):
+                    continue
+            else:
+                if neg_pattern.search(a):
+                    print("---", temp)
+                    continue
+        yield temp
+        r = os.path.split(temp)[0]
+        rep[r] = None
 
 
 def explore_folder_iterfile_repo(folder, log=fLOG):

src/pyquickhelper/helpgen/default_conf.py

@@ -73,7 +73,7 @@ def get_epkg_dictionary():
         'azure pipelines': 'https://azure.microsoft.com/en-us/services/devops/pipelines/',
         'Azure Pipelines': 'https://azure.microsoft.com/en-us/services/devops/pipelines/',
         'bokeh': 'https://bokeh.pydata.org/en/latest/',
-        'builderapi': 'http://www.sphinx-doc.org/en/stable/extdev/builderapi.html',
+        'builderapi': 'https://www.sphinx-doc.org/en/master/extdev/builderapi.html',
         'bz2': 'https://en.wikipedia.org/wiki/Bzip2',
         'cairosvg': 'https://github.com/Kozea/CairoSVG',
         'chrome': 'https://www.google.com/chrome/',
@@ -108,7 +108,7 @@ def get_epkg_dictionary():
         'Graphviz': 'http://www.graphviz.org/',
         'html': 'https://en.wikipedia.org/wiki/HTML',
         'HTML': 'https://en.wikipedia.org/wiki/HTML',
-        'imgmath': 'http://www.sphinx-doc.org/en/master/usage/extensions/math.html#module-sphinx.ext.imgmath',
+        'imgmath': 'https://www.sphinx-doc.org/en/master/usage/extensions/math.html#module-sphinx.ext.imgmath',
         'img2pdf': 'https://gitlab.mister-muffin.de/josch/img2pdf',
         'Inkscape': 'https://inkscape.org/',
         'InkScape': 'https://inkscape.org/',
@@ -202,18 +202,18 @@ def get_epkg_dictionary():
                    ('https://docs.scipy.org/doc/scipy/reference/generated/scipy.{0}.html', 1),
                    ('https://docs.scipy.org/doc/scipy/reference/generated/scipy.{0}.{1}.html', 2)),
         'SFTP': 'https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol',
-        'sphinx': 'http://www.sphinx-doc.org/en/stable/',
-        'Sphinx': 'http://www.sphinx-doc.org/en/stable/',
-        'sphinx.ext.autodoc': 'http://www.sphinx-doc.org/en/stable/ext/autodoc.html#module-sphinx.ext.autodoc',
+        'sphinx': 'https://www.sphinx-doc.org/en/master/',
+        'Sphinx': 'https://www.sphinx-doc.org/en/master/',
+        'sphinx.ext.autodoc': 'https://www.sphinx-doc.org/en/master/ext/autodoc.html#module-sphinx.ext.autodoc',
         'sphinx.ext.intersphinx': 'https://www.sphinx-doc.org/en/master/usage/extensions/intersphinx.html',
         'sphinx-gallery': 'https://sphinx-gallery.readthedocs.io/en/latest/',
-        'Sphinx application': 'http://www.sphinx-doc.org/en/stable/_modules/sphinx/application.html',
+        'Sphinx application': 'https://www.sphinx-doc.org/en/master/_modules/sphinx/application.html',
         'svg': 'https://fr.wikipedia.org/wiki/Scalable_Vector_Graphics',
         'SVG': 'https://fr.wikipedia.org/wiki/Scalable_Vector_Graphics',
         'SVN': 'https://subversion.apache.org/',
         'svn': 'https://subversion.apache.org/',
         'tar.gz': 'https://en.wikipedia.org/wiki/Tar_(computing)',
-        'toctree': 'http://www.sphinx-doc.org/en/stable/markup/toctree.html',
+        'toctree': 'https://www.sphinx-doc.org/en/master/markup/toctree.html',
         'TexnicCenter': 'http://www.texniccenter.org/',
         'tinycss2': 'http://pythonhosted.org/tinycss2/',
         'tkinter': 'https://docs.python.org/3/library/tkinter.html',
@@ -270,16 +270,16 @@ def set_sphinx_variables(fileconf, module_name, author, year, theme, theme_path,
                                         sphinx-bootstrap-theme/README.html>`_
     @param      description_latex       description latex
     @param      use_mathjax             set up the documentation to use mathjax,
-                                        see `sphinx.ext.mathjax <http://sphinx-doc.org/ext/math.html
-                                        ?highlight=math#module-sphinx.ext.mathjax>`_,
+                                        see `sphinx.ext.mathjax
+                                        <https://www.sphinx-doc.org/en/master/usage/extensions/math.html>`_,
                                         default option is True
     @param      use_lunrsearch          suggest autocompletion in sphinx,
                                         see `sphinxcontrib-lunrsearch <https://github.com/rmcgibbo/
                                         sphinxcontrib-lunrsearch>`_
     @param      enable_disabled_parts   @see fn remove_undesired_part_for_documentation
     @param      sharepost               add share button to share blog post on usual networks
     @param      custom_style            custom style sheet
-    @param      extlinks                parameter `extlinks <http://www.sphinx-doc.org/en/stable/ext/extlinks.html#confval-extlinks>`_,
+    @param      extlinks                parameter `extlinks <https://www.sphinx-doc.org/en/master/ext/extlinks.html#confval-extlinks>`_,
                                         example: ``{'issue': ('https://github.com/sdpython/pyquickhelper/issues/%s', 'issue ')}``
     @param      github_user             git(hub) user
     @param      github_repo             git(hub) project
@@ -614,7 +614,7 @@ def set_sphinx_variables(fileconf, module_name, author, year, theme, theme_path,
     # texinfo_show_urls = 'footnote'
 
     # it modifies the set of things to display inside the sidebar
-    # see http://www.sphinx-doc.org/en/stable/config.html#confval-html_sidebars
+    # see https://www.sphinx-doc.org/en/master/config.html#confval-html_sidebars
     html_sidebars = {
         '[!blog]**': ['searchbox.html', 'moduletoc.html', 'relations.html', 'sourcelink.html', ],
         'blog/**': ['searchbox.html', 'blogtoc.html', 'localtoc.html', 'sourcelink.html', ],
@@ -903,7 +903,8 @@ def get_first_line(filename):
 def _skip(app, what, name, obj, skip, options):
     """
     To skip some functions,
-    see `Skipping members <http://sphinx-doc.org/ext/autodoc.html#event-autodoc-skip-member>`_.
+    see `Skipping members
+    <https://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html>`_.
     """
     if name.startswith("_") and name not in \
             ["__qualname__",