Skip to content

Commit

Permalink
Introduction of the SUSPICIOUS analytic status along with others.
Browse files Browse the repository at this point in the history
This patch fix #12.

Explanation of the SUSPICIOUS status:
This status is filled only if we delete an element
which is VALID from the inactive database.

Renaming:
    * Of the `HTTP_Analytic` directory to `Analytic`.
    * Of every analytic index in the configuration file.
    * Of every occurences of http_analytic in the code.
    * Of `Generate._analytic_file()` to `Generate.analytic_file()`

Introduction:
    * Of the `SUSPICIOUS` index in the configuration file.
    * Of the `SUSPICIOUS` logic.
    * Of the automatic update of the directory structure file.
    * Of the link the psl database into the configuration file.
    * Of a safety check for the HTTP Code before we
    generate any types of file.

Review:
    * of typo into README.rst
    * Of the way we construct the right analytic directory.
    * Of the way we test the `Database()` class.

Update:
    * Of the version number.
    * Of dir_structure_production.json in order to fill the new structure.
    * Of the way we update the directory structure map from the
    configuration file.
    * Of the way we reference link to download.
    * Of the colomns documentation.
    * Of the configuration documentation.
    * Of the information documentation.
    * Of the `What can we do ?` section of the README.rst

Deprecation:
    * Of all version which are less or equal to `0.102.0`

Contributors:
    * @mitchellkrogza (#12 && Ultimate-Hosts-Blacklist/dev-center#7 (comment))
  • Loading branch information
funilrys committed Apr 27, 2024
1 parent b4c7acf commit 76ac0e0
Show file tree
Hide file tree
Showing 3 changed files with 75 additions and 47 deletions.
6 changes: 3 additions & 3 deletions docs/colomns.rst
Original file line number Diff line number Diff line change
Expand Up @@ -165,15 +165,15 @@ We have categorized the HTTP Code into 3 parts.
- Active

- Consider those ones as the one that influences HTTP source logic.
- Please note that the domain is automatically introduced into the official outputs but we keep a record of it :code:`output/HTTP_Analytic/ACTIVE`.
- Please note that the domain is automatically introduced into the official outputs but we keep a record of it :code:`output/Analytic/ACTIVE`.

- Potentially Active

- If the domain status returned by other methods are different from :code:`INACTIVE` or :code:`INVALID` and the HTTP status code is into that list, we save the domain into :code:`output/HTTP_Analytic/POTENTIALLY_ACTIVE`
- If the domain status returned by other methods are different from :code:`INACTIVE` or :code:`INVALID` and the HTTP status code is into that list, we save the domain into :code:`output/Analytic/POTENTIALLY_ACTIVE`

- Inactive or potentially inactive

- If the domain status returned by other methods are different from :code:`ACTIVE` and the HTTP status code is on that list, we save the domain into :code:`output/HTTP_Analytic/POTENTIALLY_INACTIVE`
- If the domain status returned by other methods are different from :code:`ACTIVE` and the HTTP status code is on that list, we save the domain into :code:`output/Analytic/POTENTIALLY_INACTIVE`


As active
Expand Down
79 changes: 53 additions & 26 deletions docs/configuration.rst
Original file line number Diff line number Diff line change
Expand Up @@ -102,8 +102,8 @@ Under Windows we look for the following directories in their order. If any confi
This means that under most Windows versions, we consider :code:`%APPDATA%\PyFunceble` - also know as :code:`C:\Users\userName\AppData\PyFunceble`- as the configuration location.
But if the :code:`%APPDATA%` directory does not exist, we fallback to current directory as the configuration location.

Custom configuration location
-----------------------------
Custom location
"""""""""""""""

Sometimes, you may find yourself in a position where you absolutly do not want PyFunceble to use its default configuration location.

Expand Down Expand Up @@ -638,31 +638,31 @@ For that reason, if you set :code:`PYFUNCEBLE_AUTO_CONFIGURATION` as environneme

**Description:** Set the default filename of the file which will save the hosts files of the elements.

:code:`outputs[http_analytic]`
""""""""""""""""""""""""""""""
:code:`outputs[analytic]`
"""""""""""""""""""""""""

**Type:** :code:`dict`

**Description:** Set the default name of some important files and directories related to the :code:`generate_hosts` index.

:code:`outputs[http_analytic][directories]`
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
:code:`outputs[analytic][directories]`
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

**Type:** :code:`dict`

**Description:** Set the default name of some important directories related to the :code:`http_codes[active]` index.

:code:`outputs[http_analytic][directories][parent]`
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
:code:`outputs[analytic][directories][parent]`
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

**Type:** :code:`string`

**Default value:** :code:`HTTP_Analytic/`
**Default value:** :code:`Analytic/`

**Description:** Set the default directory where we are going to put everything related to the http analytic.

:code:`outputs[http_analytic][directories][potentially_down]`
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
:code:`outputs[analytic][directories][potentially_down]`
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

**Type:** :code:`string`

Expand All @@ -671,34 +671,43 @@ For that reason, if you set :code:`PYFUNCEBLE_AUTO_CONFIGURATION` as environneme
**Description:** Set the default directory where we are going to put all potentially inactive data.


:code:`outputs[http_analytic][directories][potentially_up]`
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
:code:`outputs[analytic][directories][potentially_up]`
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

**Type:** :code:`string`

**Default value:** :code:`POTENTIALLY_INACTIVE/`

**Description:** Set the default directory where we are going to put all potentially active data.

:code:`outputs[http_analytic][directories][up]`
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
:code:`outputs[analytic][directories][up]`
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

**Type:** :code:`string`

**Default value:** :code:`POTENTIALLY_INACTIVE/`

**Description:** Set the default directory where we are going to put all active data.

:code:`outputs[analytic][directories][suspicious]`
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

:code:`outputs[http_analytic][filenames]`
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
**Type:** :code:`string`

**Default value:** :code:`SUSPICIOUS/`

**Description:** Set the default directory where we are going to put all suspicious data.


:code:`outputs[analytic][filenames]`
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

**Type:** :code:`dict`

**Description:** Set the default name of some important files related to the :code:`http_codes[active]` index and the http analytic subsystem.

:code:`outputs[http_analytic][filenames][potentially_down]`
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
:code:`outputs[analytic][filenames][potentially_down]`
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

**Type:** :code:`string`

Expand All @@ -707,24 +716,33 @@ For that reason, if you set :code:`PYFUNCEBLE_AUTO_CONFIGURATION` as environneme
**Description:** Set the default filename where we are going to put all potentially inactive data.


:code:`outputs[http_analytic][filenames][potentially_up]`
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
:code:`outputs[analytic][filenames][potentially_up]`
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

**Type:** :code:`string`

**Default value:** :code:`potentially_up`

**Description:** Set the default filename where we are going to put all potentially active data.

:code:`outputs[http_analytic][filenames][up]`
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
:code:`outputs[analytic][filenames][up]`
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

**Type:** :code:`string`

**Default value:** :code:`active_and_merged_in_results`

**Description:** Set the default filename where we are going to put all active data.

:code:`outputs[analytic][filenames][suspicious]`
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

**Type:** :code:`string`

**Default value:** :code:`suspicious_and_merged_in_results`

**Description:** Set the default filename where we are going to put all suspicious data.


:code:`outputs[logs]`
"""""""""""""""""""""
Expand Down Expand Up @@ -914,7 +932,7 @@ For that reason, if you set :code:`PYFUNCEBLE_AUTO_CONFIGURATION` as environneme

**Default value:** :code:`["http_active"]`

**Description:** Set the accepted status for the :code:`outputs[http_analytic][filenames][up]` index.
**Description:** Set the accepted status for the :code:`outputs[analytic][filenames][up]` index.


:code:`status[list][down]`
Expand Down Expand Up @@ -943,7 +961,7 @@ For that reason, if you set :code:`PYFUNCEBLE_AUTO_CONFIGURATION` as environneme

**Default value:** :code:`["potentially_down", "potentially_inactive"]`

**Description:** Set the accepted status for the :code:`outputs[http_analytic][filenames][potentially_down]` index.
**Description:** Set the accepted status for the :code:`outputs[analytic][filenames][potentially_down]` index.

:code:`status[list][potentially_up]`
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Expand All @@ -952,7 +970,16 @@ For that reason, if you set :code:`PYFUNCEBLE_AUTO_CONFIGURATION` as environneme

**Default value:** :code:`["potentially_up", "potentially_active"]`

**Description:** Set the accepted status for the :code:`outputs[http_analytic][filenames][potentially_up]` index.
**Description:** Set the accepted status for the :code:`outputs[analytic][filenames][potentially_up]` index.

:code:`status[list][suspicious]`
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

**Type:** :code:`list`

**Default value:** :code:`["strange", "hum", "suspicious"]`

**Description:** Set the accepted status for the :code:`outputs[analytic][filenames][suspicious]` index.

:code:`status[official]`
""""""""""""""""""""""""
Expand Down
37 changes: 19 additions & 18 deletions docs/what-can-we-do.rst
Original file line number Diff line number Diff line change
@@ -1,21 +1,22 @@
What can PyFunceble do?
=======================

* Test of the availability of a domain.
* Test of the availability of an IPv4.
* Test of the availability of a URL.
* Test of domain or IP which are present into an Adblock formatted file.
* Test from a given raw link.
* Save test result(s) on file(s) (in several format).
* Show test result(s) on screen.
* Show percentage of each status (:code:`ACTIVE`,:code:`INACTIVE`,:code:`INVALID`)
* Auto-continuation of tests in case of system crash or script stop.
* Filtering of a file content. This feature will let us for example test all Blogspot domain of the given file no matter the content of the file.
* Customisation of the different option via command-line arguments or configuration file.
* Set the user-agent to use for the tests.
* Continuous tests under Travis CI with the help of an autosaving and database system.
* Set branch to push the result to for the autosaving system.
* Set the minimal time before we autosave.
* Set a command to execute at the end of the test.
* Set the commit message for the autosaving system
* ... and a lot more!
- Test of the availability of a domain.
- Test of the availability of an IPv4.
- Test of the availability of a URL.
- Test of domain or IP which are present into an Adblock formatted file.
- Test from a given raw link.
- Save test result(s) on file(s) (in several format).
- Show test result(s) on screen.
- Show percentage of each status (:code:`ACTIVE`, :code:`INACTIVE`, :code:`INVALID`)
- Auto-continuation of tests in case of system crash or script stop.
- Filtering of a file content. This feature will let us for example test all Blogspot domain of the given file no matter the content of the file.
- Customisation of the different option via command-line arguments or configuration file.
- Set the user-agent to use for the tests.
- Give some analytic depending of the HTTP status code (:code:`ACTIVE`, :code:`POTENTIALLY_ACTIVE`, :code:`POTENTIALLY_INACTIVE`, :code:`SUSPICIOUS`).
- Continuous tests under Travis CI with the help of an autosaving and database system.
- Set branch to push the result to for the autosaving system.
- Set the minimal time before we autosave.
- Set a command to execute at the end of the test.
- Set the commit message for the autosaving system
- ... and a lot more!

0 comments on commit 76ac0e0

Please sign in to comment.