Syntax test of RPZ complaiant zone records(file) #86

spirillen · 2020-04-15T10:39:38Z

Is your feature request related to a problem? Please describe.
As I just do a --syntax test on my rpz.mypdns.org zone file I can see a problem....

The challenge is that a RPZ record not have the same rules for how to be syn-taxed as RFC:1034, it have it's own style which looks almost identically to a standard zone, but yet not.

As the test is going on, i see more and more valid Response Policy Zone records as INVALID

*.econda-monitor.de
*.top.ge
wikimedia.org.rpz-passthru
upload.wikimedia.org.rpz-passthru
*.go2app.org
*.truehits.in.th
*.wct.link
*.toplist.sk
*.adskeeper.co.uk
*.grapeshot.co.uk

These records are valid (beside the two with rpz-passthru.) they should have been listed in the pointer column.

Describe the solution you'd like
I request you to add a new switch --rpz which is compliant with the draft-vixie-dns-rpz-04

Describe alternatives you've considered
None, currently

Additional context
I have made a Easier to read and understand of how RPZ records should be formed here https://www.mypdns.org/w/rpz/#the-quot-passthru-quot-action

With the --rpz switch it should also be possible to validate an entire RPZ zone file (you will love this on your own DNS setups 😃 ) to quickly find any typos, or other malformed records that might occur from a program.

--rpz could/should support the following sources

--rpz -f = Local file
--rpz --something (hostname|IPv(4|6))(:port) = take the zone from the remote DNS server.

The current test is done with pyfunceble 3.0.8.dev (Teal Blauwbok)

The text was updated successfully, but these errors were encountered:

funilrys · 2020-04-16T13:43:34Z

Did you saw 609d506 ?

funilrys · 2020-04-16T13:45:39Z

Note to self

We should just improve the logic behind the previously mentionned commit in order to implement the RPZ specifig check.

We could also make --rpz an alias of --wildcard as the --wildcard argument was thought for that use case.

spirillen · 2020-04-16T20:27:54Z

hrm hrm.... thought i could remember something about that 😭 but as noticed i couldn't find it.

I think the wildcard should stay as it is a valid AUTH zone record type, where RPZ differs in some ways, especially when it comes to the -ip cases https://www.mypdns.org/w/rpz/#the-quot-response-ip-address

32.53.0.0.127.rpz-ip              CNAME rpz-drop. ;(Drop all queries to IP 127.0.0.53)
24.0.0.0.127.rpz-client-ip        CNAME rpz-drop. ;(Drop all requests from clients in network 127.0.0.1/24)
32.2.3.4.10.rpz-ip                CNAME .         ;(Return NXDOMAIN response to all queries that would have lead to the IP `10.4.3.2`)
48.zz.101.db8.2001.rpz-client-ip  CNAME rpz-passthru.

@spirillen

This patch fixes #86. Also: * Fix typo. Contributors: * @spirillen

funilrys · 2020-09-13T11:36:47Z

@spirillen Can you test d603982 before I merge it into the dev branch ?

spirillen · 2020-09-14T09:49:46Z

I can give it a try... But I'm kinda confused why you left out the testing of rpz-passthru.s

Give me a bit of time.

Test results from the lab

Tested directly in Debian Buster (10) (conda was not used)

Results = failure is the only option 😃

Installed by:

pip3 install --user git+https://github.com/funilrys/PyFunceble@rpz-syntax-check

Test file

drill axfr rpz.mypdns.cloud @axfr.mypdns.cloud >> rpz.mypdns.cloud # IPv6 only
drill axfr rpz.mypdns.cloud @axfr.ipv4.mypdns.cloud -p 5306 >> rpz.mypdns.cloud # IPv4 supported on port 5306

Test string:

/home/$user/.local/bin/pyfunceble --syntax --rpz -f rpz.mypdns.cloud

Test result:

Traceback (most recent call last):
  File "/home/$user/.local/bin/pyfunceble", line 8, in <module>
    sys.exit(tool())
  File "/home/$user/.local/lib/python3.7/site-packages/PyFunceble/cli/__init__.py", line 1226, in tool
    raise exception
  File "/home/$user/.local/lib/python3.7/site-packages/PyFunceble/cli/__init__.py", line 1115, in tool
    PyFunceble.CONFIGURATION.rpz = preset.switch("rpz")
  File "/home/$user/.local/lib/python3.7/site-packages/PyFunceble/config/preset.py", line 165, in switch
    raise Exception(to_print % (repr(variable), PyFunceble.LINKS.repo + "/issues."))
Exception: Impossible to switch 'rpz'. Please post an issue to https://github.com/funilrys/PyFunceble/issues.

However, even as PyF requested to update the .PyFunceble_production.yaml the following code was missing from .PyFunceble_production.yaml

# Enable/Disable the support of RPZ policies while checking for syntax (only).
rpz: False

Test 2 global installation

Same test environment, but this time I've install by

sudo -H pip3 install git+https://github.com/funilrys/PyFunceble@rpz-syntax-check

pyfunceble --version
pyfunceble 3.3.2.dev (Teal Blauwbok: Centipede)

Appended rpz: False to the .config/PyFunceble/.PyFunceble_production.yaml manually to make sure it is present.

The test run

pyfunceble --syntax --rpz -f rpz.mypdns.cloud                         
A configuration key is missing or a new version is available.
Try to merge upstream configuration file into '/home/$user/.config/PyFunceble/.PyFunceble.yaml'? [y/n] y

The following errors occured

Your version: 3.3.2.dev (Teal Blauwbok: Centipede)
Upstream version: 3.3.3.dev (Teal Blauwbok: Tick)

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 159, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw)
  File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 57, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/lib/python3.7/socket.py", line 748, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 343, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 841, in _validate_conn
    conn.connect()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 301, in connect
    conn = self._new_conn()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 168, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7fb60107e320>: Failed to establish a new connection: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 398, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='pyfunceble-not-resolved', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fb60107e320>: Failed to establish a new connection: [Errno -2] Name or service not known'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/pyfunceble", line 8, in <module>
    sys.exit(tool())
  File "/usr/local/lib/python3.7/dist-packages/PyFunceble/cli/__init__.py", line 1226, in tool
    raise exception
  File "/usr/local/lib/python3.7/dist-packages/PyFunceble/cli/__init__.py", line 1218, in tool
    generate_all_results_only=args.generate_all_files_from_database,
  File "/usr/local/lib/python3.7/dist-packages/PyFunceble/cli/dispatcher.py", line 132, in __init__
    file_path, generate_results_only, generate_all_results_only
  File "/usr/local/lib/python3.7/dist-packages/PyFunceble/cli/dispatcher.py", line 170, in dispatch_file_test
    PyFunceble.output.Constructor()
  File "/usr/local/lib/python3.7/dist-packages/PyFunceble/output/constructor.py", line 92, in __init__
    self.restore()
  File "/usr/local/lib/python3.7/dist-packages/PyFunceble/output/constructor.py", line 535, in restore
    structure = self._get_structure()
  File "/usr/local/lib/python3.7/dist-packages/PyFunceble/output/constructor.py", line 450, in _get_structure
    PyFunceble.LINKS.dir_structure.replace("master", "dev")
  File "/usr/local/lib/python3.7/dist-packages/PyFunceble/lookup/requests.py", line 289, in get
    result = self.session.get(url, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 546, in get
    return self.request('GET', url, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/PyFunceble/lookup/requests.py", line 162, in send
    response = super().send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='pyfunceble-not-resolved', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fb60107e320>: Failed to establish a new connection: [Errno -2] Name or service not known'))

Conda test env

Comming up, but quessing you have something to start with here...

spirillen · 2020-09-14T11:44:54Z

Conda test env results

Same test box as previously, using global ($HOME/).config/PyFunceble/.PyFunceble_production.yaml

conda activate "${pyfunceblePackageName}"
pyfunceble --syntax --rpz -f /tmp/rpz.mypdns.cloud

Your version: 3.3.2.dev (Teal Blauwbok: Centipede)
Upstream version: 3.3.3.dev (Teal Blauwbok: Tick)

Subject                                                                                              Status      Source    
---------------------------------------------------------------------------------------------------- ----------- ----------
3600                                                                                                 INVALID     SYNTAX    
in                                                                                                   INVALID     SYNTAX    
soa                                                                                                  INVALID     SYNTAX    
auth1.fi.matrix.rocks.                                                                               VALID       SYNTAX    
need.to.know.only.                                                                                   INVALID     SYNTAX    
2020091304                                                                                           INVALID     SYNTAX    
60                                                                                                   INVALID     SYNTAX    
604800                                                                                               INVALID     SYNTAX    
345600                                                                                               INVALID     SYNTAX    
in                                                                                                   INVALID     SYNTAX    
a                                                                                                    INVALID     SYNTAX    
116.203.32.67                                                                                        VALID       SYNTAX    
in                                                                                                   INVALID     SYNTAX    
a                                                                                                    INVALID     SYNTAX    
95.216.209.53                                                                                        VALID       SYNTAX    
600                                                                                                  INVALID     SYNTAX    
in                                                                                                   INVALID     SYNTAX    
ns                                                                                                   INVALID     SYNTAX    
ns0.dns.matrix.rocks.                                                                                VALID       SYNTAX    
in                                                                                                   INVALID     SYNTAX    
ns                                                                                                   INVALID     SYNTAX    
ns1.dns.matrix.rocks.                                                                                VALID       SYNTAX    
in                                                                                                   INVALID     SYNTAX    
aaaa                                                                                                 INVALID     SYNTAX    
2a01:4f8:1c0c:5f61::53                                                                               VALID       SYNTAX    
in                                                                                                   INVALID     SYNTAX    
aaaa                                                                                                 INVALID     SYNTAX    
2a01:4f9:c010:410e::53                                                                               VALID       SYNTAX    
172800                                                                                               INVALID     SYNTAX    
in                                                                                                   INVALID     SYNTAX    
cname                                                                                                INVALID     SYNTAX    
.                                                                                                    INVALID     SYNTAX    
in                                                                                                   INVALID     SYNTAX    
cname                                                                                                INVALID     SYNTAX    
in                                                                                                   INVALID     SYNTAX    
cname                                                                                                INVALID     SYNTAX    
in                                                                                                   INVALID     SYNTAX    
cname                                                                                                INVALID     SYNTAX    
in                                                                                                   INVALID     SYNTAX

head /tmp/rpz.mypdns.cloud 
rpz.mypdns.cloud.       3600    IN      SOA     auth1.fi.matrix.rocks. need.to.know.only. 2020091304 3600 60 604800 60
rpz.mypdns.cloud.       345600  IN      A       116.203.32.67
rpz.mypdns.cloud.       345600  IN      A       95.216.209.53
rpz.mypdns.cloud.       600     IN      NS      ns0.dns.matrix.rocks.
rpz.mypdns.cloud.       600     IN      NS      ns1.dns.matrix.rocks.
rpz.mypdns.cloud.       345600  IN      AAAA    2a01:4f8:1c0c:5f61::53
rpz.mypdns.cloud.       345600  IN      AAAA    2a01:4f9:c010:410e::53
ccp.ac.rpz.mypdns.cloud.        172800  IN      CNAME   .
*.ccp.ac.rpz.mypdns.cloud.      172800  IN      CNAME   .
directory.ac.rpz.mypdns.cloud.  172800  IN      CNAME   .

What happens here is you are first testing the SOA record, NEXT you are testing all IN CNAME . AND non of the records 😒

UPDATE
I was solely thinking domains now, of course we need to test the full record according to the syntax linked above...

Sorry for the bash version, but would for sure like to see the Python version 😈

@spirillen

This patch touches and fixes #86. Contributors: * @spirillen

funilrys · 2020-09-19T11:18:24Z

@spirillen it's a separate branch so a lot will go wrong with the configuration! But once in dev or master everything will run smoothly!

Can you test my latest improvement?

spirillen · 2020-09-19T14:49:34Z

Same test environment as previously (Debian 10, Buster)

First I test by installing the @rpz-syntax-check with the --user switch

pyfunceble --version

pyfunceble 3.3.3.dev (Teal Blauwbok: Gander)

Test file...

drill axfr rpz.mypdns.cloud @axfr.mypdns.cloud >> rpz.mypdns.cloud # IPv6 only
drill axfr rpz.mypdns.cloud @axfr.ipv4.mypdns.cloud -p 5306 >> rpz.mypdns.cloud # IPv4 supported on port 5306

Test strings

/home/$user/.local/bin/pyfunceble --syntax --rpz -f rpz.mypdns.cloud

pyfunceble --syntax --rpz -f rpz.mypdns.cloud

Rpz variable in .PyFunceble_production.yaml

grep -i 'rpz' $HOME/.config/PyFunceble/.PyFunceble_production.yaml
rpz: False

Test 1 --user installation

pip3 install --user -U git+https://github.com/funilrys/PyFunceble@rpz-syntax-check
Collecting git+https://github.com/funilrys/PyFunceble@rpz-syntax-check
  Cloning https://github.com/funilrys/PyFunceble (to revision rpz-syntax-check) to ./pip-req-build-nws6jm56
Requirement already satisfied, skipping upgrade: PyMySQL in /usr/local/lib/python3.7/dist-packages (from PyFunceble-dev==3.3.3) (0.9.3)
Requirement already satisfied, skipping upgrade: PyYAML in /usr/local/lib/python3.7/dist-packages (from PyFunceble-dev==3.3.3) (5.3.1)
Requirement already satisfied, skipping upgrade: alembic in $HOME/.local/lib/python3.7/site-packages (from PyFunceble-dev==3.3.3) (1.4.3)
Requirement already satisfied, skipping upgrade: colorama in /usr/local/lib/python3.7/dist-packages (from PyFunceble-dev==3.3.3) (0.4.3)
Requirement already satisfied, skipping upgrade: cryptography in /usr/lib/python3/dist-packages (from PyFunceble-dev==3.3.3) (2.6.1)
Requirement already satisfied, skipping upgrade: dnspython in /usr/local/lib/python3.7/dist-packages (from PyFunceble-dev==3.3.3) (1.16.0)
Requirement already satisfied, skipping upgrade: domain2idna in /usr/local/lib/python3.7/dist-packages (from PyFunceble-dev==3.3.3) (1.9.1)
Requirement already satisfied, skipping upgrade: inflection in $HOME/.local/lib/python3.7/site-packages (from PyFunceble-dev==3.3.3) (0.5.1)
Requirement already satisfied, skipping upgrade: python-box[all] in /usr/local/lib/python3.7/dist-packages (from PyFunceble-dev==3.3.3) (4.2.3)
  WARNING: python-box 4.2.3 does not provide the extra 'all'
Requirement already satisfied, skipping upgrade: python-dotenv in /usr/local/lib/python3.7/dist-packages (from PyFunceble-dev==3.3.3) (0.13.0)
Requirement already satisfied, skipping upgrade: requests<3 in /usr/lib/python3/dist-packages (from PyFunceble-dev==3.3.3) (2.21.0)
Requirement already satisfied, skipping upgrade: setuptools in /usr/lib/python3/dist-packages (from PyFunceble-dev==3.3.3) (40.8.0)
Requirement already satisfied, skipping upgrade: sqlalchemy in $HOME/.local/lib/python3.7/site-packages (from PyFunceble-dev==3.3.3) (1.3.19)
Requirement already satisfied, skipping upgrade: Mako in $HOME/.local/lib/python3.7/site-packages (from alembic->PyFunceble-dev==3.3.3) (1.1.3)
Requirement already satisfied, skipping upgrade: python-editor>=0.3 in $HOME/.local/lib/python3.7/site-packages (from alembic->PyFunceble-dev==3.3.3) (1.0.4)
Requirement already satisfied, skipping upgrade: python-dateutil in $HOME/.local/lib/python3.7/site-packages (from alembic->PyFunceble-dev==3.3.3) (2.8.1)
Requirement already satisfied, skipping upgrade: toml in /usr/local/lib/python3.7/dist-packages (from python-box[all]->PyFunceble-dev==3.3.3) (0.10.0)
Requirement already satisfied, skipping upgrade: ruamel.yaml in /usr/local/lib/python3.7/dist-packages (from python-box[all]->PyFunceble-dev==3.3.3) (0.16.10)
Requirement already satisfied, skipping upgrade: MarkupSafe>=0.9.2 in $HOME/.local/lib/python3.7/site-packages (from Mako->alembic->PyFunceble-dev==3.3.3) (1.1.1)
Requirement already satisfied, skipping upgrade: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil->alembic->PyFunceble-dev==3.3.3) (1.12.0)
Requirement already satisfied, skipping upgrade: ruamel.yaml.clib>=0.1.2; platform_python_implementation == "CPython" and python_version < "3.9" in /usr/local/lib/python3.7/dist-packages (from ruamel.yaml->python-box[all]->PyFunceble-dev==3.3.3) (0.2.0)
Building wheels for collected packages: PyFunceble-dev
  Building wheel for PyFunceble-dev (setup.py) ... done
  Created wheel for PyFunceble-dev: filename=PyFunceble_dev-3.3.3-py3-none-any.whl size=333752 sha256=f89285b4d0130fed58f753f95b7e25222bae6e4a2880a71150251e7fcf07a1b3
  Stored in directory: /tmp/pip-ephem-wheel-cache-shhh3y4j/wheels/3c/53/e0/88775abcdbc2d7b6f8689506120c7072041feedec7ad2c6cc7
Successfully built PyFunceble-dev
Installing collected packages: PyFunceble-dev
Successfully installed PyFunceble-dev-3.3.3

Test 1 --user results

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 159, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw)
  File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 57, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/lib/python3.7/socket.py", line 748, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 343, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 841, in _validate_conn
    conn.connect()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 301, in connect
    conn = self._new_conn()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 168, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7fbbdf874438>: Failed to establish a new connection: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 398, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='pyfunceble-not-resolved', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fbbdf874438>: Failed to establish a new connection: [Errno -2] Name or service not known'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "$HOME/.local/bin/pyfunceble", line 8, in <module>
    sys.exit(tool())
  File "$HOME/.local/lib/python3.7/site-packages/PyFunceble/cli/__init__.py", line 1226, in tool
    raise exception
  File "$HOME/.local/lib/python3.7/site-packages/PyFunceble/cli/__init__.py", line 1218, in tool
    generate_all_results_only=args.generate_all_files_from_database,
  File "$HOME/.local/lib/python3.7/site-packages/PyFunceble/cli/dispatcher.py", line 132, in __init__
    file_path, generate_results_only, generate_all_results_only
  File "$HOME/.local/lib/python3.7/site-packages/PyFunceble/cli/dispatcher.py", line 170, in dispatch_file_test
    PyFunceble.output.Constructor()
  File "$HOME/.local/lib/python3.7/site-packages/PyFunceble/output/constructor.py", line 92, in __init__
    self.restore()
  File "$HOME/.local/lib/python3.7/site-packages/PyFunceble/output/constructor.py", line 535, in restore
    structure = self._get_structure()
  File "$HOME/.local/lib/python3.7/site-packages/PyFunceble/output/constructor.py", line 450, in _get_structure
    PyFunceble.LINKS.dir_structure.replace("master", "dev")
  File "$HOME/.local/lib/python3.7/site-packages/PyFunceble/lookup/requests.py", line 289, in get
    result = self.session.get(url, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 546, in get
    return self.request('GET', url, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "$HOME/.local/lib/python3.7/site-packages/PyFunceble/lookup/requests.py", line 162, in send
    response = super().send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='pyfunceble-not-resolved', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fbbdf874438>: Failed to establish a new connection: [Errno -2] Name or service not known'))

Test 2 global installation

Install string

sudo -H pip3 install -U git+https://github.com/funilrys/PyFunceble@rpz-syntax-check

Collecting git+https://github.com/funilrys/PyFunceble@rpz-syntax-check
  Cloning https://github.com/funilrys/PyFunceble (to revision rpz-syntax-check) to ./pip-req-build-dd2ow06g
Requirement already satisfied, skipping upgrade: PyMySQL in /usr/local/lib/python3.7/dist-packages (from PyFunceble-dev==3.3.3) (0.9.3)
Requirement already satisfied, skipping upgrade: PyYAML in /usr/local/lib/python3.7/dist-packages (from PyFunceble-dev==3.3.3) (5.3.1)
Requirement already satisfied, skipping upgrade: alembic in /usr/local/lib/python3.7/dist-packages (from PyFunceble-dev==3.3.3) (1.4.3)
Requirement already satisfied, skipping upgrade: colorama in /usr/local/lib/python3.7/dist-packages (from PyFunceble-dev==3.3.3) (0.4.3)
Requirement already satisfied, skipping upgrade: cryptography in /usr/lib/python3/dist-packages (from PyFunceble-dev==3.3.3) (2.6.1)
Requirement already satisfied, skipping upgrade: dnspython in /usr/local/lib/python3.7/dist-packages (from PyFunceble-dev==3.3.3) (1.16.0)
Requirement already satisfied, skipping upgrade: domain2idna in /usr/local/lib/python3.7/dist-packages (from PyFunceble-dev==3.3.3) (1.9.1)
Requirement already satisfied, skipping upgrade: inflection in /usr/local/lib/python3.7/dist-packages (from PyFunceble-dev==3.3.3) (0.5.1)
Requirement already satisfied, skipping upgrade: python-box[all] in /usr/local/lib/python3.7/dist-packages (from PyFunceble-dev==3.3.3) (4.2.3)
  WARNING: python-box 4.2.3 does not provide the extra 'all'
Requirement already satisfied, skipping upgrade: python-dotenv in /usr/local/lib/python3.7/dist-packages (from PyFunceble-dev==3.3.3) (0.13.0)
Requirement already satisfied, skipping upgrade: requests<3 in /usr/lib/python3/dist-packages (from PyFunceble-dev==3.3.3) (2.21.0)
Requirement already satisfied, skipping upgrade: setuptools in /usr/lib/python3/dist-packages (from PyFunceble-dev==3.3.3) (40.8.0)
Requirement already satisfied, skipping upgrade: sqlalchemy in /usr/local/lib/python3.7/dist-packages (from PyFunceble-dev==3.3.3) (1.3.19)
Requirement already satisfied, skipping upgrade: python-dateutil in /usr/local/lib/python3.7/dist-packages (from alembic->PyFunceble-dev==3.3.3) (2.8.1)
Requirement already satisfied, skipping upgrade: Mako in /usr/local/lib/python3.7/dist-packages (from alembic->PyFunceble-dev==3.3.3) (1.1.3)
Requirement already satisfied, skipping upgrade: python-editor>=0.3 in /usr/local/lib/python3.7/dist-packages (from alembic->PyFunceble-dev==3.3.3) (1.0.4)
Requirement already satisfied, skipping upgrade: toml in /usr/local/lib/python3.7/dist-packages (from python-box[all]->PyFunceble-dev==3.3.3) (0.10.0)
Requirement already satisfied, skipping upgrade: ruamel.yaml in /usr/local/lib/python3.7/dist-packages (from python-box[all]->PyFunceble-dev==3.3.3) (0.16.10)
Requirement already satisfied, skipping upgrade: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil->alembic->PyFunceble-dev==3.3.3) (1.12.0)
Requirement already satisfied, skipping upgrade: MarkupSafe>=0.9.2 in /usr/local/lib/python3.7/dist-packages (from Mako->alembic->PyFunceble-dev==3.3.3) (1.1.1)
Requirement already satisfied, skipping upgrade: ruamel.yaml.clib>=0.1.2; platform_python_implementation == "CPython" and python_version < "3.9" in /usr/local/lib/python3.7/dist-packages (from ruamel.yaml->python-box[all]->PyFunceble-dev==3.3.3) (0.2.0)
Building wheels for collected packages: PyFunceble-dev
  Building wheel for PyFunceble-dev (setup.py) ... done
  Created wheel for PyFunceble-dev: filename=PyFunceble_dev-3.3.3-py3-none-any.whl size=333752 sha256=92c5496cff0e61975ab6204a9af5c9287653b1574693ed5e2a19045075a1c89c
  Stored in directory: /tmp/pip-ephem-wheel-cache-i5gqayrd/wheels/3c/53/e0/88775abcdbc2d7b6f8689506120c7072041feedec7ad2c6cc7
Successfully built PyFunceble-dev
Installing collected packages: PyFunceble-dev
Successfully installed PyFunceble-dev-3.3.3

pyfunceble --version
pyfunceble 3.3.3.dev (Teal Blauwbok: Gander)

Result

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 159, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw)
  File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 57, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/lib/python3.7/socket.py", line 748, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 343, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 841, in _validate_conn
    conn.connect()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 301, in connect
    conn = self._new_conn()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 168, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7ff3d26ef4e0>: Failed to establish a new connection: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 398, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='pyfunceble-not-resolved', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7ff3d26ef4e0>: Failed to establish a new connection: [Errno -2] Name or service not known'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/pyfunceble", line 8, in <module>
    sys.exit(tool())
  File "/usr/local/lib/python3.7/dist-packages/PyFunceble/cli/__init__.py", line 1226, in tool
    raise exception
  File "/usr/local/lib/python3.7/dist-packages/PyFunceble/cli/__init__.py", line 1218, in tool
    generate_all_results_only=args.generate_all_files_from_database,
  File "/usr/local/lib/python3.7/dist-packages/PyFunceble/cli/dispatcher.py", line 132, in __init__
    file_path, generate_results_only, generate_all_results_only
  File "/usr/local/lib/python3.7/dist-packages/PyFunceble/cli/dispatcher.py", line 170, in dispatch_file_test
    PyFunceble.output.Constructor()
  File "/usr/local/lib/python3.7/dist-packages/PyFunceble/output/constructor.py", line 92, in __init__
    self.restore()
  File "/usr/local/lib/python3.7/dist-packages/PyFunceble/output/constructor.py", line 535, in restore
    structure = self._get_structure()
  File "/usr/local/lib/python3.7/dist-packages/PyFunceble/output/constructor.py", line 450, in _get_structure
    PyFunceble.LINKS.dir_structure.replace("master", "dev")
  File "/usr/local/lib/python3.7/dist-packages/PyFunceble/lookup/requests.py", line 289, in get
    result = self.session.get(url, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 546, in get
    return self.request('GET', url, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/PyFunceble/lookup/requests.py", line 162, in send
    response = super().send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='pyfunceble-not-resolved', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7ff3d26ef4e0>: Failed to establish a new connection: [Errno -2] Name or service not known'))

Conda Test

Conda setup

#!/usr/bin/env bash
condaInstallDir="${HOME}/miniconda"
pyfunceblePackageName="pyfunceble-rpz"

source "${condaInstallDir}/etc/profile.d/conda.sh"

hash conda
conda update -q conda

conda activate "${pyfunceblePackageName}"

pip install --upgrade pip -q
pip uninstall -yq pyfunceble pyfunceble-dev

pip install --no-cache-dir --upgrade 'git+https://github.com/funilrys/PyFunceble@rpz-syntax-check'

python --version		# Python 3.8.5
pyfunceble --version	# pyfunceble 3.3.3.dev (Teal Blauwbok: Gander)
pip list

export PYFUNCEBLE_AUTO_CONFIGURATION=yes

pyfunceble --syntax --rpz -f /tmp/rpz.mypdns.cloud

Results

Subject                                              Status      Source    
---------------------------------------------------- ----------- ----------
rpz.mypdns.cloud.                                    VALID       SYNTAX    
ccp.ac.rpz.mypdns.cloud.                             VALID       SYNTAX    
*.ccp.ac.rpz.mypdns.cloud.                           VALID       SYNTAX    
directory.ac.rpz.mypdns.cloud.                       VALID       SYNTAX    
fix.ac.rpz.mypdns.cloud.                             VALID       SYNTAX    
gay.fix.ac.rpz.mypdns.cloud.                         VALID       SYNTAX    
forum.ac.rpz.mypdns.cloud.                           VALID       SYNTAX    
scarlatinastudyo.forum.ac.rpz.mypdns.cloud.          VALID       SYNTAX    
www.scarlatinastudyo.forum.ac.rpz.mypdns.cloud.      VALID       SYNTAX    
kimono.ac.rpz.mypdns.cloud.                          VALID       SYNTAX    
www.kimono.ac.rpz.mypdns.cloud.                      VALID       SYNTAX    
link.ac.rpz.mypdns.cloud.                            VALID       SYNTAX    
*.link.ac.rpz.mypdns.cloud.                          VALID       SYNTAX    
www.link.ac.rpz.mypdns.cloud.                        VALID       SYNTAX    
primewire.ac.rpz.mypdns.cloud.                       VALID       SYNTAX    
www.primewire.ac.rpz.mypdns.cloud.                   VALID       SYNTAX    
putlocker.ac.rpz.mypdns.cloud.                       VALID       SYNTAX    
www.putlocker.ac.rpz.mypdns.cloud.                   VALID       SYNTAX    
sms.ac.rpz.mypdns.cloud.                             VALID       SYNTAX    
reg.sms.ac.rpz.mypdns.cloud.                         VALID       SYNTAX    
theacademy.ac.rpz.mypdns.cloud.                      VALID       SYNTAX    
www.theacademy.ac.rpz.mypdns.cloud.                  VALID       SYNTAX    
tube.ac.rpz.mypdns.cloud.                            VALID       SYNTAX    
*.tube.ac.rpz.mypdns.cloud.                          VALID       SYNTAX    
twit.ac.rpz.mypdns.cloud.                            VALID       SYNTAX    
upload.ac.rpz.mypdns.cloud.                          VALID       SYNTAX    
*.upload.ac.rpz.mypdns.cloud.                        VALID       SYNTAX    
apple-ios.academy.rpz.mypdns.cloud.                  VALID       SYNTAX    
www.apple-ios.academy.rpz.mypdns.cloud.              VALID       SYNTAX    
...
^C
Thanks for using PyFunceble!
Share your experience on Twitter with #PyFunceble!
Have a feedback, an issue or an improvement idea?
Let us know on GitHub!

Conda record test (Not syntax)

I'm doing this test as it appears (tl;dr;) that you have implanted it in ee5e89e

Your implementation of this is a good choice and you are actually sparing me a supplement to the feature request to Several input sources at once -f -uf -ad, where I would suggest a new read-in of zone files directly, IE. as -(-)?zone and/or -(-)?axfr as the zone file could be tested from a local or NFS located file.
This would benefits your self to, for testing your new public DNS server As you can use the AXFR feature directly. I recommend using drill from at least console drill axfr zone.name @auth.server as dig happens to complain a lot (+noids +ids etc. as RPZ isn't RFC standard zone layouts')

Test string

pyfunceble --rpz -f /tmp/adult.mypdns.cloud # As rpz.mypdns.cloud is to big for a full test

Comment

From the on-screen layout, it can appear that you are not stripping the zone from the test records... and this can lead to a misconception of what is test and how.

These records are valid, however they was marked 'INVALID`, and found a few more alike.
(From onscreen output)

annuairesexeporno.com.adult.mypdns.cloud.            ACTIVE      XXX       
*.annuairesexeporno.com.adult.mypdns.cloud.          INVALID     XXX       

antiquetwat.com.adult.mypdns.cloud.                  ACTIVE      XXX       
*.antiquetwat.com.adult.mypdns.cloud.                INVALID     XXX

Test result

This is a complement to the above comment

Status      Percentage   Numbers     
----------- ------------ ------------
ACTIVE      97%          2377        
INACTIVE    1%           35          
INVALID     1%           29

tree --prune -f output/
output
├── output/continue.json
├── output/hosts
│   ├── output/hosts/ACTIVE
│   │   └── output/hosts/ACTIVE/hosts
│   ├── output/hosts/INACTIVE
│   │   └── output/hosts/INACTIVE/hosts
│   └── output/hosts/INVALID
│       └── output/hosts/INVALID/hosts
├── output/logs
│   └── output/logs/percentage
│       └── output/logs/percentage/percentage.txt
└── output/splited
    ├── output/splited/ACTIVE
    ├── output/splited/INACTIVE
    └── output/splited/INVALID

head output/hosts/INVALID/hosts
# Generated by PyFunceble (v3.3.3.dev) / https://git.io/vpZoI
# Date of generation: 2020-09-19T13:31:43.402102

0.0.0.0 *.onlyrealzoo.club.adult.mypdns.cloud.
0.0.0.0 *.igayporn.tv.sds.rncdn7.co.adult.mypdns.cloud.
0.0.0.0 *.amateursfuckporn.com.adult.mypdns.cloud.

head output/hosts/ACTIVE/hosts 
# Generated by PyFunceble (v3.3.3.dev) / https://git.io/vpZoI
# Date of generation: 2020-09-19T13:29:34.868796

0.0.0.0 adult.mypdns.cloud.
0.0.0.0 *.adult.adult.mypdns.cloud.
0.0.0.0 yespornplease.adult.adult.mypdns.cloud.
0.0.0.0 *.00webcams.com.adult.mypdns.cloud.
0.0.0.0 *.0xxx.ws.adult.mypdns.cloud.
0.0.0.0 *.1cam.webcam.adult.mypdns.cloud.
0.0.0.0 *.1watchmygf.com.adult.mypdns.cloud.

head output/hosts/INACTIVE/hosts 
# Generated by PyFunceble (v3.3.3.dev) / https://git.io/vpZoI
# Date of generation: 2020-09-19T13:31:29.186047

0.0.0.0 greentoons.club.adult.mypdns.cloud.
0.0.0.0 adultadworld.com.adult.mypdns.cloud.
0.0.0.0 amateurxfilms.com.adult.mypdns.cloud.
0.0.0.0 badgirlsblog.com.adult.mypdns.cloud.
0.0.0.0 beautifulhairypussy.com.adult.mypdns.cloud.
0.0.0.0 bestjavporn.com.adult.mypdns.cloud.
0.0.0.0 bimbolive.com.adult.mypdns.cloud.

Test with `-m`

Test string

pyfunceble -h -m --rpz -f /tmp/adult.mypdns.cloud

Test results

tree --prune -f output/
output
├── output/continue.json
├── output/logs
│   └── output/logs/percentage
│       └── output/logs/percentage/percentage.txt
└── output/splited
    ├── output/splited/ACTIVE
    ├── output/splited/INACTIVE
    └── output/splited/INVALID

Test string

pyfunceble -h --plain -m --rpz -f /tmp/adult.mypdns.cloud

tree --prune -f output/
output
├── output/continue.json
├── output/domains
│   └── output/domains/ACTIVE
│       └── output/domains/ACTIVE/list
├── output/logs
│   └── output/logs/percentage
│       └── output/logs/percentage/percentage.txt
└── output/splited
    └── output/splited/ACTIVE

Conclusion

It appears that the way you have implemented the urllib3 is broken in python versions < 3.8.5.

keybase://chat/spirillen,funilrys/1131 & keybase://chat/spirillen,funilrys/1132

The overall impression is that the PR is getting close to a working complement

However there are clearly a few hiccups that needs to be addressed.

The --rpz test isn't stripping the the zone.name (in this case the .adult.mypdns.cloud.) from the test string
1. However the combined --syntax --rpz shall keep the full string fr validating the zone.
2. Since the zone.name isn't stripped, you can't test real records or your RPZ Firewall filters.
The default generated output is in ("useless") hosts formatted style. I would personally have expected domain style as default output, as the input source is "domain" styled and not hosts file, But there are of curse several ways to use this program, and the default output have always been hosts formatted files.

A Thought

This test have giving me a thought I'll like to share with the rest of you nice people.

What should be the default output if -h is set in the test string?

spirillen · 2020-09-19T21:56:13Z

Note to self:
RPZ test file: https://www.mypdns.org/paste/raw/23/

@spirillen

This patch touches #86. Contributors: * @spirillen

@spirillen

This patch touches #86. Indeed, before this patch, we were converting (when needed) wildcard and RPZ only inside the `PyFunceble.Check` class. From now, the `PyFunceble.Check` class will not do any conversion. All the conversion (wildcard, rpz) are now made on a deeper level. This change let us have access to the conversion (wildcard, rpz) with any kind of test (availability, syntax, reputation, etc.). Contributors: * @spirillen

spirillen · 2020-09-20T16:38:15Z

Test string:

pyfunceble -h -db --database-type mariadb -f /var/storage01/repos/github/mypdns/porn-records/submit_here/adult.mypdns.cloud/wildcard.list --dns 192.168.1.105 --mining -m -ex -a -nf --complements --share-logs

Error output

Traceback (most recent call last):
File "/home/joakim/miniconda/envs/pyfunceble-dev/lib/python3.8/site-packages/PyFunceble/core/multiprocess.py", line 91, in run
Process.run(self)
File "/home/joakim/miniconda/envs/pyfunceble-dev/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/joakim/miniconda/envs/pyfunceble-dev/lib/python3.8/site-packages/PyFunceble/core/multiprocess.py", line 191, in test
result = APICore(
File "/home/joakim/miniconda/envs/pyfunceble-dev/lib/python3.8/site-packages/PyFunceble/core/api.py", line 216, in availability
return self.domain_and_ip()
File "/home/joakim/miniconda/envs/pyfunceble-dev/lib/python3.8/site-packages/PyFunceble/core/api.py", line 294, in domain_and_ip
CLICore.save_into_database(data, self.db_file_name)
File "/home/joakim/miniconda/envs/pyfunceble-dev/lib/python3.8/site-packages/PyFunceble/core/cli.py", line 212, in save_into_database
.filter(Status.tested == output["given"])
KeyError: 'given'

Test string:

pyfunceble -h -db --database-type mariadb -f /var/storage01/repos/github/mypdns/porn-records/submit_here/adult.mypdns.cloud/wildcard.list --dns 192.168.1.105 -m -ex -a -nf --complements --share-logs

Error output

Traceback (most recent call last):
File "/home/joakim/miniconda/envs/pyfunceble-dev/lib/python3.8/site-packages/PyFunceble/core/multiprocess.py", line 91, in run
Process.run(self)
File "/home/joakim/miniconda/envs/pyfunceble-dev/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/joakim/miniconda/envs/pyfunceble-dev/lib/python3.8/site-packages/PyFunceble/core/multiprocess.py", line 191, in test
result = APICore(
File "/home/joakim/miniconda/envs/pyfunceble-dev/lib/python3.8/site-packages/PyFunceble/core/api.py", line 216, in availability
return self.domain_and_ip()
File "/home/joakim/miniconda/envs/pyfunceble-dev/lib/python3.8/site-packages/PyFunceble/core/api.py", line 294, in domain_and_ip
CLICore.save_into_database(data, self.db_file_name)
File "/home/joakim/miniconda/envs/pyfunceble-dev/lib/python3.8/site-packages/PyFunceble/core/cli.py", line 212, in save_into_database
.filter(Status.tested == output["given"])
KeyError: 'given'

Test string:

pyfunceble -h -db --database-type mariadb -f /var/storage01/repos/github/mypdns/porn-records/submit_here/adult.mypdns.cloud/wildcard.list --dns 192.168.1.105 -m -ex -a

Unexspected prints of dots...

Test string:

pyfunceble -h -db --database-type mariadb -f /var/storage01/repos/github/mypdns/porn-records/submit_here/adult.mypdns.cloud/wildcard.list --dns 192.168.1.105 -m -ex

Error output

Subject Status HTTP Code

Traceback (most recent call last):
File "/home/joakim/miniconda/envs/pyfunceble-dev/bin/pyfunceble", line 8, in
sys.exit(tool())
File "/home/joakim/miniconda/envs/pyfunceble-dev/lib/python3.8/site-packages/PyFunceble/cli/init.py", line 1222, in tool
raise exception
File "/home/joakim/miniconda/envs/pyfunceble-dev/lib/python3.8/site-packages/PyFunceble/cli/init.py", line 1207, in tool
Dispatcher(
File "/home/joakim/miniconda/envs/pyfunceble-dev/lib/python3.8/site-packages/PyFunceble/cli/dispatcher.py", line 131, in init
self.dispatch_file_test(
File "/home/joakim/miniconda/envs/pyfunceble-dev/lib/python3.8/site-packages/PyFunceble/cli/dispatcher.py", line 174, in dispatch_file_test
PyFunceble.core.Multiprocess(file_path, "domain").run_test()
File "/home/joakim/miniconda/envs/pyfunceble-dev/lib/python3.8/site-packages/PyFunceble/core/multiprocess.py", line 603, in run_test
self.cleanup(self.autocontinue, self.autosave, test_completed=True)
File "/home/joakim/miniconda/envs/pyfunceble-dev/lib/python3.8/site-packages/PyFunceble/core/file.py", line 464, in cleanup
self.generate_files()
File "/home/joakim/miniconda/envs/pyfunceble-dev/lib/python3.8/site-packages/PyFunceble/core/file.py", line 239, in generate_files
self.generate_files_of_status(
File "/home/joakim/miniconda/envs/pyfunceble-dev/lib/python3.8/site-packages/PyFunceble/core/file.py", line 200, in generate_files_of_status
data.given,
AttributeError: 'Status' object has no attribute 'given'

funilrys · 2020-09-21T13:52:26Z

@spirillen 5628bf3 fixes the KeyError: 'given' issue.

I'm now looking at the issue with the multiprocessing.

@spirillen

This patch touches #86. Contributors: * @spirillen

funilrys · 2020-09-21T14:29:16Z

This issue is ready for dev deployment. It's waiting for your next report @spirillen.

spirillen · 2020-09-22T12:19:54Z

well that one would be rather short....

as you can see on keybase the pyfuncele_statuswas growen to 4.8GB of dubplicates, so I decided to reload a fresh copy of the original DB...

pip uninstall -yq pyfunceble pyfunceble-dev
pip install --no-cache-dir --upgrade pyfunceble-dev

OR

pip uninstall -yq pyfunceble pyfunceble-dev
pip install --no-cache-dir --upgrade 'git+https://github.com/funilrys/PyFunceble@dev'

OR

pip uninstall -yq pyfunceble pyfunceble-dev
pip install --no-cache-dir --upgrade 'git+https://github.com/funilrys/PyFunceble@rpz-syntax-check'

Test string: (With the --rpz for this branch)

pyfunceble -db --database-type mariadb --syntax -m -a -f /tmp/adult.mypdns.cloud

Same error:

Subject                                                                                              Status      Expiration Date   Source     Analyze Date        
---------------------------------------------------------------------------------------------------- ----------- ----------------- ---------- --------------------
Traceback (most recent call last):
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
    self.dialect.do_execute(
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 593, in do_execute
    cursor.execute(statement, parameters)
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/cursors.py", line 163, in execute
    result = self._query(query)
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/cursors.py", line 321, in _query
    conn.query(q)
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/connections.py", line 505, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/connections.py", line 724, in _read_query_result
    result.read()
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/connections.py", line 1069, in read
    first_packet = self.connection._read_packet()
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/connections.py", line 676, in _read_packet
    packet.raise_for_error()
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/protocol.py", line 223, in raise_for_error
    err.raise_mysql_exception(self._data)
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
    raise errorclass(errno, errval)
pymysql.err.ProgrammingError: (1146, "Table 'PyFunceble.pyfunceble_status' doesn't exist")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/bin/pyfunceble", line 8, in <module>
    sys.exit(tool())
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/PyFunceble/cli/__init__.py", line 1209, in tool
    raise exception
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/PyFunceble/cli/__init__.py", line 1194, in tool
    Dispatcher(
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/PyFunceble/cli/dispatcher.py", line 131, in __init__
    self.dispatch_file_test(
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/PyFunceble/cli/dispatcher.py", line 174, in dispatch_file_test
    PyFunceble.core.Multiprocess(file_path, "domain").run_test()
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/PyFunceble/core/multiprocess.py", line 566, in run_test
    self.__run_multiprocess_test(shadow_file, manager, tracker=tracker)
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/PyFunceble/core/multiprocess.py", line 422, in __run_multiprocess_test
    self.__start_process(
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/PyFunceble/core/multiprocess.py", line 303, in __start_process
    if subject in self.autocontinue:
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/PyFunceble/engine/auto_continue.py", line 152, in __contains__
    db_session.query(Status)
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3463, in one
    ret = self.one_or_none()
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3432, in one_or_none
    ret = list(self)
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3508, in __iter__
    return self._execute_and_instances(context)
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3533, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
    return meth(self, multiparams, params)
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1124, in _execute_clauseelement
    ret = self._execute_context(
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1316, in _execute_context
    self._handle_dbapi_exception(
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1510, in _handle_dbapi_exception
    util.raise_(
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
    self.dialect.do_execute(
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 593, in do_execute
    cursor.execute(statement, parameters)
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/cursors.py", line 163, in execute
    result = self._query(query)
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/cursors.py", line 321, in _query
    conn.query(q)
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/connections.py", line 505, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/connections.py", line 724, in _read_query_result
    result.read()
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/connections.py", line 1069, in read
    first_packet = self.connection._read_packet()
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/connections.py", line 676, in _read_packet
    packet.raise_for_error()
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/protocol.py", line 223, in raise_for_error
    err.raise_mysql_exception(self._data)
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
    raise errorclass(errno, errval)
sqlalchemy.exc.ProgrammingError: (pymysql.err.ProgrammingError) (1146, "Table 'PyFunceble.pyfunceble_status' doesn't exist")
[SQL: SELECT pyfunceble_status.id AS pyfunceble_status_id, pyfunceble_status.created AS pyfunceble_status_created, pyfunceble_status.modified AS pyfunceble_status_modified, pyfunceble_status.file_id AS pyfunceble_status_file_id, pyfunceble_status.tested AS pyfunceble_status_tested, pyfunceble_status._status AS pyfunceble_status__status, pyfunceble_status.status AS pyfunceble_status_status, pyfunceble_status._status_source AS pyfunceble_status__status_source, pyfunceble_status.status_source AS pyfunceble_status_status_source, pyfunceble_status.domain_syntax_validation AS pyfunceble_status_domain_syntax_validation, pyfunceble_status.expiration_date AS pyfunceble_status_expiration_date, pyfunceble_status.http_status_code AS pyfunceble_status_http_status_code, pyfunceble_status.ipv4_range_syntax_validation AS pyfunceble_status_ipv4_range_syntax_validation, pyfunceble_status.ipv4_syntax_validation AS pyfunceble_status_ipv4_syntax_validation, pyfunceble_status.ipv6_range_syntax_validation AS pyfunceble_status_ipv6_range_syntax_validation, pyfunceble_status.ipv6_syntax_validation AS pyfunceble_status_ipv6_syntax_validation, pyfunceble_status.subdomain_syntax_validation AS pyfunceble_status_subdomain_syntax_validation, pyfunceble_status.url_syntax_validation AS pyfunceble_status_url_syntax_validation, pyfunceble_status.is_complement AS pyfunceble_status_is_complement, pyfunceble_status.test_completed AS pyfunceble_status_test_completed, pyfunceble_status.tested_at AS pyfunceble_status_tested_at 
FROM pyfunceble_status INNER JOIN pyfunceble_file ON pyfunceble_file.id = pyfunceble_status.file_id 
WHERE pyfunceble_status.tested = %(tested_1)s AND pyfunceble_file.path = %(path_1)s]
[parameters: {'tested_1': '3600', 'path_1': '/tmp/adult.mypdns.cloud'}]
(Background on this error at: http://sqlalche.me/e/13/f405)

And here is something a bit more weird...

Clean DB, no tables..

pip uninstall -yq pyfunceble pyfunceble-dev
WARNING: Skipping pyfunceble as it is not installed.
:/tmp$ pip install --no-cache-dir --upgrade 'git+https://github.com/funilrys/PyFunceble@dev'

Error

Collecting git+https://github.com/funilrys/PyFunceble@dev
  Cloning https://github.com/funilrys/PyFunceble (to revision dev) to ./pip-req-build-jbh7bd53
Requirement already satisfied, skipping upgrade: alembic in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from PyFunceble-dev==3.3.3) (1.4.3)
Requirement already satisfied, skipping upgrade: colorama in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from PyFunceble-dev==3.3.3) (0.4.3)
Requirement already satisfied, skipping upgrade: cryptography in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from PyFunceble-dev==3.3.3) (3.1)
Requirement already satisfied, skipping upgrade: dnspython in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from PyFunceble-dev==3.3.3) (2.0.0)
Requirement already satisfied, skipping upgrade: domain2idna in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from PyFunceble-dev==3.3.3) (1.12.0)
Requirement already satisfied, skipping upgrade: inflection in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from PyFunceble-dev==3.3.3) (0.5.1)
Requirement already satisfied, skipping upgrade: PyMySQL in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from PyFunceble-dev==3.3.3) (0.10.1)
Requirement already satisfied, skipping upgrade: python-box[all] in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from PyFunceble-dev==3.3.3) (5.1.1)
Requirement already satisfied, skipping upgrade: python-dotenv in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from PyFunceble-dev==3.3.3) (0.14.0)
Requirement already satisfied, skipping upgrade: PyYAML in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from PyFunceble-dev==3.3.3) (5.3.1)
Requirement already satisfied, skipping upgrade: requests<3 in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from PyFunceble-dev==3.3.3) (2.24.0)
Requirement already satisfied, skipping upgrade: setuptools in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from PyFunceble-dev==3.3.3) (49.6.0.post20200814)
Requirement already satisfied, skipping upgrade: sqlalchemy in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from PyFunceble-dev==3.3.3) (1.3.19)
Requirement already satisfied, skipping upgrade: Mako in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from alembic->PyFunceble-dev==3.3.3) (1.1.3)
Requirement already satisfied, skipping upgrade: python-dateutil in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from alembic->PyFunceble-dev==3.3.3) (2.8.1)
Requirement already satisfied, skipping upgrade: python-editor>=0.3 in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from alembic->PyFunceble-dev==3.3.3) (1.0.4)
Requirement already satisfied, skipping upgrade: six>=1.4.1 in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from cryptography->PyFunceble-dev==3.3.3) (1.15.0)
Requirement already satisfied, skipping upgrade: cffi!=1.11.3,>=1.8 in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from cryptography->PyFunceble-dev==3.3.3) (1.14.2)
Requirement already satisfied, skipping upgrade: toml; extra == "all" in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from python-box[all]->PyFunceble-dev==3.3.3) (0.10.1)
Requirement already satisfied, skipping upgrade: ruamel.yaml; extra == "all" in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from python-box[all]->PyFunceble-dev==3.3.3) (0.16.12)
Requirement already satisfied, skipping upgrade: msgpack; extra == "all" in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from python-box[all]->PyFunceble-dev==3.3.3) (1.0.0)
Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from requests<3->PyFunceble-dev==3.3.3) (2020.6.20)
Requirement already satisfied, skipping upgrade: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from requests<3->PyFunceble-dev==3.3.3) (1.25.10)
Requirement already satisfied, skipping upgrade: idna<3,>=2.5 in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from requests<3->PyFunceble-dev==3.3.3) (2.10)
Requirement already satisfied, skipping upgrade: chardet<4,>=3.0.2 in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from requests<3->PyFunceble-dev==3.3.3) (3.0.4)
Requirement already satisfied, skipping upgrade: MarkupSafe>=0.9.2 in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from Mako->alembic->PyFunceble-dev==3.3.3) (1.1.1)
Requirement already satisfied, skipping upgrade: pycparser in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from cffi!=1.11.3,>=1.8->cryptography->PyFunceble-dev==3.3.3) (2.20)
Requirement already satisfied, skipping upgrade: ruamel.yaml.clib>=0.1.2; platform_python_implementation == "CPython" and python_version < "3.9" in /home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages (from ruamel.yaml; extra == "all"->python-box[all]->PyFunceble-dev==3.3.3) (0.2.2)
Building wheels for collected packages: PyFunceble-dev
  Building wheel for PyFunceble-dev (setup.py) ... done
  Created wheel for PyFunceble-dev: filename=PyFunceble_dev-3.3.3-py3-none-any.whl size=320945 sha256=a6e60709481d21cd77e91482300d11f62cd5404bb1752ba97a4f017ff218cba9
  Stored in directory: /tmp/pip-ephem-wheel-cache-9rw_8j2s/wheels/49/58/3a/77d097367acd934fcfa7ff9e67a79eed3dd9610929f3b5a4fe
Successfully built PyFunceble-dev
Installing collected packages: PyFunceble-dev
Successfully installed PyFunceble-dev-3.3.3


pyfunceble -db --database-type mariadb --syntax -m -a -f /tmp/adult.mypdns.cloud     

██████╗ ██╗   ██╗███████╗██╗   ██╗███╗   ██╗ ██████╗███████╗██████╗ ██╗     ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║   ██║████╗  ██║██╔════╝██╔════╝██╔══██╗██║     ██╔════╝
██████╔╝ ╚████╔╝ █████╗  ██║   ██║██╔██╗ ██║██║     █████╗  ██████╔╝██║     █████╗
██╔═══╝   ╚██╔╝  ██╔══╝  ██║   ██║██║╚██╗██║██║     ██╔══╝  ██╔══██╗██║     ██╔══╝
██║        ██║   ██║     ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝        ╚═╝   ╚═╝      ╚═════╝ ╚═╝  ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝



Subject                                                                                              Status      Expiration Date   Source     Analyze Date        
---------------------------------------------------------------------------------------------------- ----------- ----------------- ---------- --------------------
Traceback (most recent call last):
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
    self.dialect.do_execute(
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 593, in do_execute
    cursor.execute(statement, parameters)
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/cursors.py", line 163, in execute
    result = self._query(query)
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/cursors.py", line 321, in _query
    conn.query(q)
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/connections.py", line 505, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/connections.py", line 724, in _read_query_result
    result.read()
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/connections.py", line 1069, in read
    first_packet = self.connection._read_packet()
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/connections.py", line 676, in _read_packet
    packet.raise_for_error()
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/protocol.py", line 223, in raise_for_error
    err.raise_mysql_exception(self._data)
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
    raise errorclass(errno, errval)
pymysql.err.ProgrammingError: (1146, "Table 'PyFunceble.pyfunceble_status' doesn't exist")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/bin/pyfunceble", line 8, in <module>
    sys.exit(tool())
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/PyFunceble/cli/__init__.py", line 1209, in tool
    raise exception
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/PyFunceble/cli/__init__.py", line 1194, in tool
    Dispatcher(
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/PyFunceble/cli/dispatcher.py", line 131, in __init__
    self.dispatch_file_test(
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/PyFunceble/cli/dispatcher.py", line 174, in dispatch_file_test
    PyFunceble.core.Multiprocess(file_path, "domain").run_test()
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/PyFunceble/core/multiprocess.py", line 566, in run_test
    self.__run_multiprocess_test(shadow_file, manager, tracker=tracker)
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/PyFunceble/core/multiprocess.py", line 422, in __run_multiprocess_test
    self.__start_process(
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/PyFunceble/core/multiprocess.py", line 303, in __start_process
    if subject in self.autocontinue:
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/PyFunceble/engine/auto_continue.py", line 152, in __contains__
    db_session.query(Status)
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3463, in one
    ret = self.one_or_none()
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3432, in one_or_none
    ret = list(self)
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3508, in __iter__
    return self._execute_and_instances(context)
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3533, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
    return meth(self, multiparams, params)
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1124, in _execute_clauseelement
    ret = self._execute_context(
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1316, in _execute_context
    self._handle_dbapi_exception(
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1510, in _handle_dbapi_exception
    util.raise_(
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
    self.dialect.do_execute(
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 593, in do_execute
    cursor.execute(statement, parameters)
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/cursors.py", line 163, in execute
    result = self._query(query)
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/cursors.py", line 321, in _query
    conn.query(q)
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/connections.py", line 505, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/connections.py", line 724, in _read_query_result
    result.read()
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/connections.py", line 1069, in read
    first_packet = self.connection._read_packet()
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/connections.py", line 676, in _read_packet
    packet.raise_for_error()
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/protocol.py", line 223, in raise_for_error
    err.raise_mysql_exception(self._data)
  File "/home/spirillen/miniconda/envs/pyfunceble-rpz/lib/python3.8/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
    raise errorclass(errno, errval)
sqlalchemy.exc.ProgrammingError: (pymysql.err.ProgrammingError) (1146, "Table 'PyFunceble.pyfunceble_status' doesn't exist")
[SQL: SELECT pyfunceble_status.id AS pyfunceble_status_id, pyfunceble_status.created AS pyfunceble_status_created, pyfunceble_status.modified AS pyfunceble_status_modified, pyfunceble_status.file_id AS pyfunceble_status_file_id, pyfunceble_status.tested AS pyfunceble_status_tested, pyfunceble_status._status AS pyfunceble_status__status, pyfunceble_status.status AS pyfunceble_status_status, pyfunceble_status._status_source AS pyfunceble_status__status_source, pyfunceble_status.status_source AS pyfunceble_status_status_source, pyfunceble_status.domain_syntax_validation AS pyfunceble_status_domain_syntax_validation, pyfunceble_status.expiration_date AS pyfunceble_status_expiration_date, pyfunceble_status.http_status_code AS pyfunceble_status_http_status_code, pyfunceble_status.ipv4_range_syntax_validation AS pyfunceble_status_ipv4_range_syntax_validation, pyfunceble_status.ipv4_syntax_validation AS pyfunceble_status_ipv4_syntax_validation, pyfunceble_status.ipv6_range_syntax_validation AS pyfunceble_status_ipv6_range_syntax_validation, pyfunceble_status.ipv6_syntax_validation AS pyfunceble_status_ipv6_syntax_validation, pyfunceble_status.subdomain_syntax_validation AS pyfunceble_status_subdomain_syntax_validation, pyfunceble_status.url_syntax_validation AS pyfunceble_status_url_syntax_validation, pyfunceble_status.is_complement AS pyfunceble_status_is_complement, pyfunceble_status.test_completed AS pyfunceble_status_test_completed, pyfunceble_status.tested_at AS pyfunceble_status_tested_at 
FROM pyfunceble_status INNER JOIN pyfunceble_file ON pyfunceble_file.id = pyfunceble_status.file_id 
WHERE pyfunceble_status.tested = %(tested_1)s AND pyfunceble_file.path = %(path_1)s]
[parameters: {'tested_1': '3600', 'path_1': '/tmp/adult.mypdns.cloud'}]
(Background on this error at: http://sqlalche.me/e/13/f405)

DB visual layout after this

as a site note I can confirm this is happening, when I try to load the sql dump, do to the keyrefs vs the read-in order directly with mariadb

With: mariadb -u $user -p -h '$host' -r PyFunceble < file_path.sql

ERROR 1005 (HY000) at line 263: Can't create table `PyFunceble`.`pyfunceble_mined` (errno: 121 "Duplicate key on write or update")

I don't know if it me, who is tired here, but it looks like at step back from previously...

funilrys · 2020-12-20T10:51:56Z

@spirillen, Can you test this with 4.0.0 ?

funilrys · 2020-12-22T23:07:16Z

Closing. Feel free to test and reopen if not fixed or if necessary.

spirillen · 2020-12-23T02:00:20Z

@spirillen, Can you test this with 4.0.0 ?

Could you make a temporary "Re-test" project and throw these into... I can't keep up.

funilrys · 2020-12-23T15:07:53Z

@spirillen Okay. Done. @\me and I'll add them in there.

spirillen · 2020-12-23T21:55:01Z

/REOPEN

idna_subject LIKE "%mypdns.cloud%"

tested with the smaller adult.mypdns.cloud

Result, --rpz are testing the full records, not the domain blocked by the record

id	created_at	modified_at	idna_subject	checker_type	destination	source	tested_at
361280	2020-12-23 21:30:08	2020-12-23 21:30:08	adult.mypdns.cloud.	AVAILABILITY	adult.list	/home/pi/Downloads/pyfunceble/adult.list	2020-12-23 21:30:07
361283	2020-12-23 21:30:09	2020-12-23 21:30:09	00webcams.com.adult.mypdns.cloud.	AVAILABILITY	adult.list	/home/pi/Downloads/pyfunceble/adult.list	2020-12-23 21:30:07
361288	2020-12-23 21:30:11	2020-12-23 21:30:11	0xxx.ws.adult.mypdns.cloud.	AVAILABILITY	adult.list	/home/pi/Downloads/pyfunceble/adult.list	2020-12-23 21:30:08
361294	2020-12-23 21:30:13	2020-12-23 21:30:13	100bucksbabes.com.adult.mypdns.cloud.	AVAILABILITY	adult.list	/home/pi/Downloads/pyfunceble/adult.list	2020-12-23 21:30:09
361299	2020-12-23 21:30:15	2020-12-23 21:30:15	101top.biz.adult.mypdns.cloud.	AVAILABILITY	adult.list	/home/pi/Downloads/pyfunceble/adult.list	2020-12-23 21:30:09
361304	2020-12-23 21:30:17	2020-12-23 21:30:17	17sexvideo.com.adult.mypdns.cloud.	AVAILABILITY	adult.list	/home/pi/Downloads/pyfunceble/adult.list	2020-12-23 21:30:10
361309	2020-12-23 21:30:19	2020-12-23 21:30:19	18-21yo.com.adult.mypdns.cloud.	AVAILABILITY	adult.list	/home/pi/Downloads/pyfunceble/adult.list	2020-12-23 21:30:10

The zone should be subtracted from the test, only validating the actually record. In this case

00webcams.com
0xxx.ws
100bucksbabes.com
101top.biz
17sexvideo.com
18-21yo.com

Still have to test with --syntax

spirillen · 2020-12-23T22:28:42Z

Results of --syntax --rpz vs expected (@\me)

Test file

cat adult.list 
adult.mypdns.cloud.     3600    IN      SOA     auth1.fi.matrix.rocks. need.to.know.only. 2020102242 300 60 604800 3600
*.00webcams.com.adult.mypdns.cloud.     86400   IN      CNeMa   . ; Error in TYPE
*.00webcams.com.adult.mypdns.net.     86400   IN      CNAME   . ; Wrong ZONE (NET not CLOUD)
*.00webcams.com.adult.mypdns.org.     86400   IN      CNAME   . ; Wrong ZONE (ORG not CLOUD)
*.00webcams.com.adult.mypdns.cloud.     86400   IN      CNAME   . : VALID as it match the ZONE

Versus the --rpz -f adult-list without the --syntax, I would expect this test to make a full validation of the zone file.
This means here should have been 3 INVALID and 1 VALID

My POV, this test test failed.

funilrys · 2020-12-23T22:42:44Z

@spirillen, I understand your POV about ; Error in TYPE but I'm not sure that PyFunceble is the tool suited for that filter. That opens the doors to all special cases and god knows how much there is today and how much is going to be asked for..

I'm ready to implement the testing without the adult.mypdns.cloud. assuming that the SOA is given beforehand. But otherwise, PyFunceble won't really care about anything after the first part of a line.

spirillen · 2020-12-23T22:44:26Z

Appendix to previous comment as I have expanded the number of test records

cat adult.list 
adult.mypdns.cloud.     3600    IN      SOA     auth1.fi.matrix.rocks. need.to.know.only. 2020102242 300 60 604800 3600
.00webcams.com.adult.mypdns.cloud.     86400   IN      CNAME   .
*.00webcams.com.rpz-passthru.adult.mypdns.cloud.     86400   IN      CNAME   .
53.0.0.127.rpz-ip               86400   IN      CNAME rpz-drop ; Missin CIDR and ending dot for rpz-drop\.
53.0.0.127.rpz-ip               86400   IN      CNAME rpz-drop. ; Missin CIDR
32.53.0.0.127.rpz-ip               86400   IN      CNAME rpz-drop ; Missin ending dot for rpz-drop\.
*.00webcams.com.adult.mypdns.cloud.     86400   IN      CNeMa   . ; Error in TYPE
*.00webcams.com.adult.mypdns.net.     86400   IN      CNAME   . ; Wrong ZONE (NET not CLOUD)
*.00webcams.com.adult.mypdns.org.     86400   IN      CNAME   . ; Wrong ZONE (ORG not CLOUD)
*.00webcams.com.adult.mypdns.cloud.     86400   IN      CNAME   . : VALID
48.zz.101.db8.2001.rpz-client-ip  CNAME rpz-passthru. ; VALID
.zz.101.db8.2001.rpz-client-ip  CNAME rpz-passthru. ; Missing CIDR and can not start with a dot
zz.101.db8.2001.rpz-client-ip  CNAME rpz-passthru. ; Missing CIDR

bash /var/storage/repos/github/PyFunceble-Templates/pyfunceble-miniconda/pyfunceble4-arm7.sh /home/pi/Downloads/adult.list 
        I'm your hungry cheese missing a piece...
        Are you ready to go hunting the red pills
        so we can hunt down evil ghosts

        Are you ready to start?

        loading kacman...
Enter output directory for test results: /tmp/pyfunceble4/2343

What version would you like to use?
1. master
2. dev
3. 4.alpha

Which version of PyFunceble would you like to use?: pyfunceble4: 3
Already on '4.0.0-dev'
Your branch is up to date with 'origin/4.0.0-dev'.
Enter any custom test string: --syntax --rpz --no-files
Would you like to use your default pyfunceble enviroment?: [Y/n] Y
pyfunceble 4.0.0a21.dev (Blue Duckling)

Fatal Error: Unexpected '/' in '127.0.0.53/32'

spirillen · 2020-12-23T22:59:26Z

@spirillen, I understand your POV about ; Error in TYPE but I'm not sure that PyFunceble is the tool suited for that filter. That opens the doors to all special cases and god knows how much there is today and how much is going to be asked for..

Let's not disagree about that, but it would be needed if we should be able to load a zone directly from a DNS server at some point. I'm open to make a discussion about this 😃

see DISC: #149

... assuming that the SOA ...
That seem impossible without 👍 We should just remember to note that in the docs....

SOA line is required to complete the test

This will actually also make it possible to test whether an RPZ is set up correct 😋 delete the SOA and hammer your DNS, now everything should be in compliance with your RPZ zone 👍

@spirillen

This patch fixes #86. Indeed, before this patch, the SOA was not split of a given RPZ policy. This patch fixes that by collecting the SOAs and then splitting them while decoding the RPZ policies. Contributors: * @spirillen

spirillen · 2020-12-27T17:48:20Z

Unfortunately I'm probably not able to test this until morning, the URL test is still running

spirillen · 2020-12-27T20:45:46Z

TEST OK

funilrys self-assigned this Apr 16, 2020

funilrys added enhancement features labels Apr 16, 2020

funilrys added this to the 3.2.0 milestone Apr 24, 2020

funilrys modified the milestones: 3.2.0, __future__ May 9, 2020

funilrys modified the milestones: __future__, 3.4.0 Sep 11, 2020

funilrys added a commit that referenced this issue Sep 11, 2020

Introduction of the --rpz argument and logic.

d603982

This patch fixes #86. Also: * Fix typo. Contributors: * @spirillen

funilrys added a commit that referenced this issue Sep 19, 2020

Improvement of the RPZ and wildcard implementation.

ee5e89e

This patch touches and fixes #86. Contributors: * @spirillen

spirillen mentioned this issue Sep 19, 2020

sqlalchemy +empty database (new) #120

Closed

funilrys added a commit that referenced this issue Sep 20, 2020

Add missing comment signs.

45e04c2

This patch touches #86. Contributors: * @spirillen

funilrys added a commit that referenced this issue Sep 21, 2020

Fix issue with the error handling while using multiple process.

8707a48

This patch touches #86. Contributors: * @spirillen

spirillen mentioned this issue Sep 22, 2020

Error when running PyFunceble in Powershell after installing with pip #127

Closed

funilrys added this to Todo in 4.0.0: Follow - Up via automation Dec 20, 2020

funilrys moved this from Todo to In progress in 4.0.0: Follow - Up Dec 20, 2020

funilrys modified the milestones: 3.4.0, 4.0.0 Dec 20, 2020

funilrys moved this from In progress to Done in 4.0.0: Follow - Up Dec 21, 2020

funilrys closed this as completed Dec 22, 2020

funilrys added this to Todo in 4.0.0.0 Re-Testing: Follow-Up via automation Dec 23, 2020

funilrys moved this from Todo to Waiting for input in 4.0.0.0 Re-Testing: Follow-Up Dec 23, 2020

funilrys reopened this Dec 23, 2020

4.0.0: Follow - Up automation moved this from Done to Todo Dec 23, 2020

4.0.0.0 Re-Testing: Follow-Up automation moved this from Waiting for input to Todo Dec 23, 2020

spirillen mentioned this issue Dec 27, 2020

FP invalid records (4.0.0a13.dev) #146

Closed

funilrys closed this as completed Dec 27, 2020

4.0.0: Follow - Up automation moved this from Todo to Done Dec 27, 2020

funilrys moved this from Todo to Waiting for input in 4.0.0.0 Re-Testing: Follow-Up Dec 27, 2020

funilrys moved this from Waiting for input to Successfully Tested in 4.0.0.0 Re-Testing: Follow-Up Dec 27, 2020

funilrys moved this from Successfully Tested to Done in 4.0.0.0 Re-Testing: Follow-Up Jan 5, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Syntax test of RPZ complaiant zone records(file) #86

Syntax test of RPZ complaiant zone records(file) #86

spirillen commented Apr 15, 2020

funilrys commented Apr 16, 2020

funilrys commented Apr 16, 2020

spirillen commented Apr 16, 2020 •

edited

funilrys commented Sep 13, 2020

spirillen commented Sep 14, 2020 •

edited

spirillen commented Sep 14, 2020 •

edited

funilrys commented Sep 19, 2020

spirillen commented Sep 19, 2020

spirillen commented Sep 19, 2020

spirillen commented Sep 20, 2020

funilrys commented Sep 21, 2020

funilrys commented Sep 21, 2020 •

edited

spirillen commented Sep 22, 2020

funilrys commented Dec 20, 2020

funilrys commented Dec 22, 2020

spirillen commented Dec 23, 2020

funilrys commented Dec 23, 2020

spirillen commented Dec 23, 2020

spirillen commented Dec 23, 2020

funilrys commented Dec 23, 2020 •

edited

spirillen commented Dec 23, 2020

spirillen commented Dec 23, 2020

spirillen commented Dec 27, 2020

spirillen commented Dec 27, 2020

Syntax test of RPZ complaiant zone records(file) #86

Syntax test of RPZ complaiant zone records(file) #86

Comments

spirillen commented Apr 15, 2020

funilrys commented Apr 16, 2020

funilrys commented Apr 16, 2020

spirillen commented Apr 16, 2020 • edited

funilrys commented Sep 13, 2020

spirillen commented Sep 14, 2020 • edited

Test results from the lab

Test 2 global installation

The test run

Conda test env

spirillen commented Sep 14, 2020 • edited

Conda test env results

funilrys commented Sep 19, 2020

spirillen commented Sep 19, 2020

pyfunceble --version

Test file...

Test strings

Rpz variable in .PyFunceble_production.yaml

Test 1 --user installation

Test 1 --user results

Test 2 global installation

Install string

Result

Conda Test

Conda setup

Results

Conda record test (Not syntax)

Test string

Comment

Test result

Test with -m

Conclusion

A Thought

spirillen commented Sep 19, 2020

spirillen commented Sep 20, 2020

funilrys commented Sep 21, 2020

funilrys commented Sep 21, 2020 • edited

spirillen commented Sep 22, 2020

funilrys commented Dec 20, 2020

funilrys commented Dec 22, 2020

spirillen commented Dec 23, 2020

funilrys commented Dec 23, 2020

spirillen commented Dec 23, 2020

spirillen commented Dec 23, 2020

funilrys commented Dec 23, 2020 • edited

spirillen commented Dec 23, 2020

spirillen commented Dec 23, 2020

spirillen commented Dec 27, 2020

spirillen commented Dec 27, 2020

spirillen commented Apr 16, 2020 •

edited

spirillen commented Sep 14, 2020 •

edited

spirillen commented Sep 14, 2020 •

edited

Test with `-m`

funilrys commented Sep 21, 2020 •

edited

funilrys commented Dec 23, 2020 •

edited