Merge pull request #1461 from p-l-/support-py311

Support Python 3.11

.github/CONTRIBUTING.md

@@ -80,7 +80,7 @@ dependency makes it harder for users to install IVRE.
 
 We try to comply with the some guidelines for new code:
 
--   The code **must** be compatible with Python 3.7 to 3.10.
+-   The code **must** be compatible with Python 3.7 to 3.11.
 
 -   The code should be PEP-8 compliant; you can check your code with
     [pep8](https://pypi.python.org/pypi/pep8).

.github/workflows/elastic.yml

@@ -24,7 +24,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: ['3.7', '3.10']
+        python-version: ['3.7', '3.11']
         mongodb-version: ['5.0']
         elastic-version: ['7.10.2']
 

.github/workflows/linting.yml

@@ -24,7 +24,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: ['3.10']
+        python-version: ['3.11']
 
     steps:
 
@@ -53,7 +53,7 @@ jobs:
     - run: pylint -e all -d abstract-method,arguments-differ,attribute-defined-outside-init,broad-except,duplicate-code,fixme,function-redefined,global-statement,import-error,invalid-name,locally-disabled,missing-docstring,no-member,protected-access,super-init-not-called,suppressed-message,too-few-public-methods,too-many-ancestors,too-many-arguments,too-many-boolean-expressions,too-many-branches,too-many-instance-attributes,too-many-lines,too-many-locals,too-many-nested-blocks,too-many-public-methods,too-many-return-statements,too-many-statements,unsubscriptable-object,unused-argument,line-too-long,consider-using-f-string ivre ./doc/conf.py ./setup.py ./bin/ivre
     - run: pylint -e all -d unused-argument,too-many-arguments,missing-function-docstring,missing-class-docstring,missing-module-docstring,multiple-statements,invalid-name,too-few-public-methods ./pkg/stubs/*.pyi
 
-    - run: isort --profile black --check-only ivre ./doc/conf.py ./setup.py ./bin/ivre ./pkg/stubs/*.pyi
+    - run: isort --profile black --check-only ivre ./doc/conf.py ./setup.py ./bin/ivre ./tests/tests.py ./pkg/stubs/*.pyi
 
     - run: mv ivre_bak ivre
     - run: MYPYPATH=./pkg/stubs/ mypy --follow-imports=skip --disallow-untyped-calls --disallow-untyped-decorators --disallow-untyped-defs --disallow-incomplete-defs --no-implicit-optional --warn-redundant-casts --warn-unused-ignores --warn-return-any ./ivre/{active,analyzer,data,parser,tools,types}/*.py ./ivre/{__init__,activecli,agent,config,flow,geoiputils,graphroute,keys,nmapopt,utils,zgrabout}.py

.github/workflows/maxmind.yml

@@ -24,7 +24,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: ['3.7', '3.10']
+        python-version: ['3.7', '3.11']
 
     steps:
 

.github/workflows/mongodb.yml

@@ -25,7 +25,7 @@ jobs:
       fail-fast: false
       matrix:
         # https://docs.mongodb.com/drivers/pymongo/#compatibility
-        python-version: ['3.7', '3.10']
+        python-version: ['3.7', '3.11']
         mongodb-version: ['3.6', '5.0']
 
     steps:

.github/workflows/postgres.yml

@@ -24,7 +24,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: ['3.7', '3.10']
+        python-version: ['3.7', '3.11']
         postgresql-version: ['12', '13']
 
     steps:

.github/workflows/sqlite.yml

@@ -24,7 +24,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: ['3.7', '3.10']
+        python-version: ['3.7', '3.11']
 
     steps:
 

.github/workflows/tinydb.yml

@@ -24,7 +24,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: ['3.7', '3.10']
+        python-version: ['3.7', '3.11']
 
     steps:
 

README.md

@@ -1,7 +1,8 @@
 [![Join the chat at Gitter](https://badges.gitter.im/ivre/ivre.svg)](https://gitter.im/ivre/ivre)
 [![Follow on Twitter](https://img.shields.io/twitter/follow/IvreRocks.svg?logo=twitter)](https://twitter.com/IvreRocks)
-[![PyPI downloads](https://pepy.tech/badge/ivre)](https://pepy.tech/project/ivre)
-![Docker pulls](https://img.shields.io/docker/pulls/ivre/web)
+[![GitHub stars](https://img.shields.io/github/stars/ivre/ivre?style=social)](https://github.com/ivre/ivre)
+[![PyPI downloads](https://pepy.tech/badge/ivre)](https://pypi.org/project/ivre/)
+[![Docker pulls](https://img.shields.io/docker/pulls/ivre/web)](https://hub.docker.com/u/ivre/)
 
 [![MongoDB tests](https://github.com/ivre/ivre/actions/workflows/mongodb.yml/badge.svg?branch=master)](https://github.com/ivre/ivre/actions/workflows/mongodb.yml/?branch=master)
 [![Elasticsearch tests](https://github.com/ivre/ivre/actions/workflows/elastic.yml/badge.svg?branch=master)](https://github.com/ivre/ivre/actions/workflows/elastic.yml/?branch=master)

doc/dev/linting.rst

@@ -35,5 +35,5 @@ GitHub actions
 Code linting and spell checking is performed in a dedicated `GitHub
 action <https://github.com/ivre/ivre/actions/workflows/linting.yml>`_
 (see :ref:`dev/tests:GitHub actions`), togethter with the Maxmind
-tests. Pylint and Codespell only run with Python 3.10, while Flake8
+tests. Pylint and Codespell only run with Python 3.11, while Flake8
 runs with all Python versions.

doc/dev/tests.rst

@@ -46,7 +46,7 @@ place your samples there (the PCAP files must have the extension
 and the Nmap JSON results must have the extension ``.json``).
 
 Then, run ``python ./tests.py`` (optionally replace ``python`` by the
-alternative interpreter you want to use, e.g., ``python3.10``; note
+alternative interpreter you want to use, e.g., ``python3.11``; note
 that ``coverage.py`` must be installed for this interpreter).  The
 first run will create a ``samples/results`` file with the expected
 values for some results. The next runs will use those values to check
@@ -87,6 +87,6 @@ GitHub actions
 
 Tests are run with several MongoDB and PostgreSQL versions, as well as
 TinyDB, SQLite and Elasticsearch for each pull requests. The tests run
-with Python 3.7 to 3.10.
+with Python 3.7 to 3.11.
 
 The configurations are in the `.github/workflows/*.yml` YAML files.

ivre/nmapopt.py

@@ -21,8 +21,8 @@
 """This sub-module is responsible for generating Nmap options."""
 
 
-import pipes
 from argparse import ArgumentParser
+from shlex import quote
 from typing import Dict, Iterable, List, Optional
 
 from ivre import config
@@ -167,4 +167,4 @@ def build_nmap_options(template: str = "default") -> List[str]:
 
 
 def build_nmap_commandline(template: str = "default") -> str:
-    return " ".join(pipes.quote(elt) for elt in build_nmap_options(template=template))
+    return " ".join(quote(elt) for elt in build_nmap_options(template=template))

ivre/tools/auditdom.py

@@ -23,9 +23,9 @@
 
 import argparse
 import json
-import pipes
 import sys
 from datetime import datetime
+from shlex import quote
 from typing import Any, Iterable, Optional
 
 from ivre import VERSION
@@ -63,7 +63,7 @@ def displayfunction(cur: Iterable[Record], scan: Optional[Any] = None) -> None:
         "version": VERSION,
         "xmloutputversion": "1.04",
         # argv[0] does not need quotes due to how it is handled by ivre
-        "args": " ".join(sys.argv[:1] + [pipes.quote(arg) for arg in sys.argv[1:]]),
+        "args": " ".join(sys.argv[:1] + [quote(arg) for arg in sys.argv[1:]]),
         "scaninfos": [
             {
                 "type": "audit DNS domain",

pkg/runchecks

@@ -35,7 +35,7 @@ pylint -e all -d abstract-method,arguments-differ,attribute-defined-outside-init
 
 pylint -e all -d unused-argument,too-many-arguments,missing-function-docstring,missing-class-docstring,missing-module-docstring,multiple-statements,invalid-name,too-few-public-methods ./pkg/stubs/*.pyi && echo "pylint stubs OK"
 
-isort --profile black --check-only ivre ./doc/conf.py ./setup.py ./bin/ivre ./pkg/stubs/*.pyi && echo "isort OK"
+isort --profile black --check-only ivre ./doc/conf.py ./setup.py ./bin/ivre ./tests/tests.py ./pkg/stubs/*.pyi && echo "isort OK"
 
 rstcheck -r ./doc/ && echo "rstcheck OK"
 if ! find ./doc/ -type f -name '*.rst' -print0 | xargs -0 grep '[[:space:]]$'; then echo "trailing spaces OK"; else echo "!! trailing spaces !!"; false; fi

setup.py

@@ -153,6 +153,7 @@ def _write_pkg_file(self, file):
         "Programming Language :: Python :: 3.8",
         "Programming Language :: Python :: 3.9",
         "Programming Language :: Python :: 3.10",
+        "Programming Language :: Python :: 3.11",
         "Topic :: Scientific/Engineering :: Information Analysis",
         "Topic :: Security",
         "Topic :: System :: Networking",

tests/tests.py

@@ -17,19 +17,12 @@
 # along with IVRE. If not, see <http://www.gnu.org/licenses/>.
 
 
-from ast import literal_eval
-from contextlib import contextmanager
-from datetime import datetime, timedelta
 import errno
-from functools import reduce
-from glob import glob
-from io import BytesIO
 import json
 import os
-import pipes
 import random
 import re
-from select import select
+import shlex
 import shutil
 import signal
 import socket
@@ -39,23 +32,29 @@
 import tempfile
 import time
 import unittest
-from urllib.request import HTTPError, Request, urlopen
+from ast import literal_eval
+from contextlib import contextmanager
+from datetime import datetime, timedelta
+from functools import reduce
+from glob import glob
+from io import BytesIO
+from select import select
 from urllib.parse import quote
-
+from urllib.request import HTTPError, Request, urlopen
 
 import ivre
+import ivre.analyzer
 import ivre.config
 import ivre.db
 import ivre.flow
 import ivre.mathutils
-import ivre.parser.zeek
 import ivre.parser.iptables
+import ivre.parser.zeek
 import ivre.passive
 import ivre.target
 import ivre.utils
 import ivre.web.utils
 import ivre.xmlnmap
-import ivre.analyzer
 
 HTTPD_PORT = 18080
 HTTPD_HOSTNAME = socket.gethostname()
@@ -168,7 +167,7 @@ def run_passiverecon_worker(bulk_mode=None):
                 "logs",
                 "--progname",
                 " ".join(
-                    pipes.quote(elt)
+                    shlex.quote(elt)
                     for elt in COVERAGE
                     + [
                         "run",

web/static/doc/_sources/dev/linting.rst.txt

@@ -35,5 +35,5 @@ GitHub actions
 Code linting and spell checking is performed in a dedicated `GitHub
 action <https://github.com/ivre/ivre/actions/workflows/linting.yml>`_
 (see :ref:`dev/tests:GitHub actions`), togethter with the Maxmind
-tests. Pylint and Codespell only run with Python 3.10, while Flake8
+tests. Pylint and Codespell only run with Python 3.11, while Flake8
 runs with all Python versions.

web/static/doc/_sources/dev/tests.rst.txt

@@ -46,7 +46,7 @@ place your samples there (the PCAP files must have the extension
 and the Nmap JSON results must have the extension ``.json``).
 
 Then, run ``python ./tests.py`` (optionally replace ``python`` by the
-alternative interpreter you want to use, e.g., ``python3.10``; note
+alternative interpreter you want to use, e.g., ``python3.11``; note
 that ``coverage.py`` must be installed for this interpreter).  The
 first run will create a ``samples/results`` file with the expected
 values for some results. The next runs will use those values to check
@@ -87,6 +87,6 @@ GitHub actions
 
 Tests are run with several MongoDB and PostgreSQL versions, as well as
 TinyDB, SQLite and Elasticsearch for each pull requests. The tests run
-with Python 3.7 to 3.10.
+with Python 3.7 to 3.11.
 
 The configurations are in the `.github/workflows/*.yml` YAML files.

web/static/doc/dev/linting.html

@@ -122,7 +122,7 @@ <h2>GitHub actions<a class="headerlink" href="#github-actions" title="Permalink
 <p>Code linting and spell checking is performed in a dedicated <a class="reference external" href="https://github.com/ivre/ivre/actions/workflows/linting.yml">GitHub
 action</a>
 (see <a class="reference internal" href="tests.html#github-actions"><span class="std std-ref">GitHub actions</span></a>), togethter with the Maxmind
-tests. Pylint and Codespell only run with Python 3.10, while Flake8
+tests. Pylint and Codespell only run with Python 3.11, while Flake8
 runs with all Python versions.</p>
 </section>
 </section>

web/static/doc/dev/tests.html

@@ -140,7 +140,7 @@ <h2>First run<a class="headerlink" href="#first-run" title="Permalink to this he
 <code class="docutils literal notranslate"><span class="pre">.pcap</span></code>, the Nmap XML result files must have the extension <code class="docutils literal notranslate"><span class="pre">.xml</span></code>,
 and the Nmap JSON results must have the extension <code class="docutils literal notranslate"><span class="pre">.json</span></code>).</p>
 <p>Then, run <code class="docutils literal notranslate"><span class="pre">python</span> <span class="pre">./tests.py</span></code> (optionally replace <code class="docutils literal notranslate"><span class="pre">python</span></code> by the
-alternative interpreter you want to use, e.g., <code class="docutils literal notranslate"><span class="pre">python3.10</span></code>; note
+alternative interpreter you want to use, e.g., <code class="docutils literal notranslate"><span class="pre">python3.11</span></code>; note
 that <code class="docutils literal notranslate"><span class="pre">coverage.py</span></code> must be installed for this interpreter).  The
 first run will create a <code class="docutils literal notranslate"><span class="pre">samples/results</span></code> file with the expected
 values for some results. The next runs will use those values to check
@@ -180,7 +180,7 @@ <h2>Failures<a class="headerlink" href="#failures" title="Permalink to this head
 <h2>GitHub actions<a class="headerlink" href="#github-actions" title="Permalink to this heading"></a></h2>
 <p>Tests are run with several MongoDB and PostgreSQL versions, as well as
 TinyDB, SQLite and Elasticsearch for each pull requests. The tests run
-with Python 3.7 to 3.10.</p>
+with Python 3.7 to 3.11.</p>
 <p>The configurations are in the <cite>.github/workflows/*.yml</cite> YAML files.</p>
 </section>
 </section>