Source code cleaning

[jtof-fap]

Adding encoding choice in library mode and enforce UTF-8 encoding by default in:

• User's input files;
• Payload files;
• Output log files.

Payload files renaming for better clarity:

renamed:    payloads/const_header_methods.lst -> payloads/header_http_methods.lst
renamed:    payloads/const_header_hosts.lst -> payloads/header_ip_hosts.lst
renamed:    payloads/const_header_ports.lst -> payloads/header_ports.lst
renamed:    payloads/const_header_schemes.lst -> payloads/header_proto_schemes.lst
renamed:    payloads/const_endpaths.lst -> payloads/internal_endpaths.lst
renamed:    payloads/const_http_methods.lst -> payloads/internal_http_methods.lst
renamed:    payloads/const_internal_ips.lst -> payloads/internal_ip_hosts.lst
renamed:    payloads/const_midpaths.lst -> payloads/internal_midpaths.lst
renamed:    payloads/const_ports.lst -> payloads/internal_ports.lst
renamed:    payloads/const_protos.lst -> payloads/internal_proto_schemes.lst

Improving error handling and logging level:

• Now, when the subdomain of the URL cannot be resolved, the URL is simply ignored and no longer causes a fatal error;
• Fixing double exceptions raised during encoding errors or when the input file does not exist;
• Harmonization of logging level from few critical to error.

Harmonize 'http_headers_scheme' payload generation style with 'http_headers_ip':

• Reintegation of Forwarded, Front-End-Https, X-Forwarded-HTTPS and X-Forwarded-SSL headers in header_proto_schemes.lst payload file.

Comparison of inner classes more generic (__eq__(self, other) function).

[jtof-fap]

Diff of bup-payloads-2022-10-04.lst file with previous version:

diff --git a/tests-history/bup-payloads-2022-09-24.lst b/tests-history/bup-payloads-2022-10-04.lst
index 658d8c6..000cb1a 100644
--- a/tests-history/bup-payloads-2022-09-24.lst
+++ b/tests-history/bup-payloads-2022-10-04.lst
@@ -142,8 +142,8 @@ Bypasser has generated 3044 payloads for 'http://127.0.0.1:8000/foo/bar' url:
 [end_paths] http://127.0.0.1:8000/foo/bar/true/
 [end_paths] http://127.0.0.1:8000/foo/bar/~
 [end_paths] http://127.0.0.1:8000/foo/bar/~/
-[end_paths] http://127.0.0.1:8000/foo/bar/°/
-[end_paths] http://127.0.0.1:8000/foo/bar/°//
+[end_paths] http://127.0.0.1:8000/foo/bar/°/
+[end_paths] http://127.0.0.1:8000/foo/bar/°//
 [end_paths] http://127.0.0.1:8000/foo/bar0
 [end_paths] http://127.0.0.1:8000/foo/bar0/
 [end_paths] http://127.0.0.1:8000/foo/bar1
@@ -166,8 +166,8 @@ Bypasser has generated 3044 payloads for 'http://127.0.0.1:8000/foo/bar' url:
 [end_paths] http://127.0.0.1:8000/foo/bar\/\//
 [end_paths] http://127.0.0.1:8000/foo/bar~
 [end_paths] http://127.0.0.1:8000/foo/bar~/
-[end_paths] http://127.0.0.1:8000/foo/bar°/
-[end_paths] http://127.0.0.1:8000/foo/bar°//
+[end_paths] http://127.0.0.1:8000/foo/bar°/
+[end_paths] http://127.0.0.1:8000/foo/bar°//
 [http_headers_ip] -H Ali-CDN-Real-IP: * http://127.0.0.1:8000/foo/bar
 [http_headers_ip] -H Ali-CDN-Real-IP: 0.0.0.0 http://127.0.0.1:8000/foo/bar
 [http_headers_ip] -H Ali-CDN-Real-IP: 0177.1 http://127.0.0.1:8000/foo/bar

[laluka]

Seems ok :)

[laluka]

What's the reason for the diff here? UTF-8 fix? Does it still have the behavior we need?

-[end_paths] http://127.0.0.1:8000/foo/bar/°//
+[end_paths] http://127.0.0.1:8000/foo/bar/°/

[jtof-fap]

UTF-8 encoding support! Yes, desired behavior, ° is in endpath-as-is ;-)